[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:11:06 dist-test-slave-21b9 krb5kdc[21095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:11:06 dist-test-slave-21b9 krb5kdc[21095](info): set up 2 sockets
Aug 12 01:11:06 dist-test-slave-21b9 krb5kdc[21095](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 01:11:07.945294 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:11:08 dist-test-slave-21b9 krb5kdc[21095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:11:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:11:08.163169 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:40153
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:40153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:08.542418 21113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:08.542732 21113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:08.542872 21113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:08.550496 21113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:08.550629 21113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:08.550688 21113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:08.550774 21113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:08.550830 21113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:11:08.561934 21113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:40153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:40153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:08.563418 21113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:08.564838 21113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:08.578008 21119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:08.577971 21120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:08.578068 21124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:08.578706 21113 server_base.cc:1034] running on GCE node
I20260812 01:11:08.579684 21113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:08.581261 21113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:08.582482 21113 hybrid_clock.cc:648] HybridClock initialized: now 1786497068582425 us; error 50 us; skew 500 ppm
I20260812 01:11:08.582844 21113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:08 dist-test-slave-21b9 krb5kdc[21095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497068, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:08.586453 21113 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:11:08.588181 21113 webserver.cc:460] Webserver started at http://127.20.151.62:39265/ using document root <none> and password file <none>
I20260812 01:11:08.588912 21113 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:08.589078 21113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:08.589416 21113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:08.592620 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "87aca3f83d7d4011b9cb10ad0cc54c1a"
format_stamp: "Formatted at 2026-08-12 01:11:08 on dist-test-slave-21b9"
server_key: "66a047aa71c8dad0baed7a3aa138f5c2"
server_key_iv: "0ff71c20d128f05015a5cb115abfd7e4"
server_key_version: "encryptionkey@0"
I20260812 01:11:08.593374 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "87aca3f83d7d4011b9cb10ad0cc54c1a"
format_stamp: "Formatted at 2026-08-12 01:11:08 on dist-test-slave-21b9"
server_key: "66a047aa71c8dad0baed7a3aa138f5c2"
server_key_iv: "0ff71c20d128f05015a5cb115abfd7e4"
server_key_version: "encryptionkey@0"
I20260812 01:11:08.598724 21113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:11:08.602710 21141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:08.604200 21113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:11:08.604380 21113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "87aca3f83d7d4011b9cb10ad0cc54c1a"
format_stamp: "Formatted at 2026-08-12 01:11:08 on dist-test-slave-21b9"
server_key: "66a047aa71c8dad0baed7a3aa138f5c2"
server_key_iv: "0ff71c20d128f05015a5cb115abfd7e4"
server_key_version: "encryptionkey@0"
I20260812 01:11:08.604574 21113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:08.629596 21113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:08.634509 21113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:08.634845 21113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:08.658677 21113 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:40153
I20260812 01:11:08.658679 21228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:40153 every 8 connection(s)
I20260812 01:11:08.660331 21113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:08.664831 21230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:08.665609 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 21113
I20260812 01:11:08.666038 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:08.668668 21084 external_mini_cluster.cc:1428] Setting key 4c8a6d805be2f0fa90c750108b12dfe8
I20260812 01:11:08.678879 21230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a: Bootstrap starting.
I20260812 01:11:08.682511 21230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:08.683951 21230 log.cc:824] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:08.687695 21230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a: No bootstrap required, opened a new log
I20260812 01:11:08.697592 21230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87aca3f83d7d4011b9cb10ad0cc54c1a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40153 } }
I20260812 01:11:08.698040 21230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:08.698168 21230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87aca3f83d7d4011b9cb10ad0cc54c1a, State: Initialized, Role: FOLLOWER
I20260812 01:11:08.698932 21230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87aca3f83d7d4011b9cb10ad0cc54c1a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40153 } }
I20260812 01:11:08.699198 21230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:08.699396 21230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:08.699594 21230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [term 0 FOLLOWER]: Advancing to term 1
Aug 12 01:11:08 dist-test-slave-21b9 krb5kdc[21095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:08.702286 21230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87aca3f83d7d4011b9cb10ad0cc54c1a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40153 } }
I20260812 01:11:08.702836 21230 leader_election.cc:302] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87aca3f83d7d4011b9cb10ad0cc54c1a; no voters:
I20260812 01:11:08.703653 21230 leader_election.cc:288] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:08.703858 21237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:08.705375 21237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [term 1 LEADER]: Becoming Leader. State: Replica: 87aca3f83d7d4011b9cb10ad0cc54c1a, State: Running, Role: LEADER
I20260812 01:11:08.706202 21237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87aca3f83d7d4011b9cb10ad0cc54c1a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40153 } }
I20260812 01:11:08.706704 21230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:08.709442 21242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87aca3f83d7d4011b9cb10ad0cc54c1a. Latest consensus state: current_term: 1 leader_uuid: "87aca3f83d7d4011b9cb10ad0cc54c1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87aca3f83d7d4011b9cb10ad0cc54c1a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40153 } } }
I20260812 01:11:08.709815 21242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:08.710640 21239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87aca3f83d7d4011b9cb10ad0cc54c1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87aca3f83d7d4011b9cb10ad0cc54c1a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40153 } } }
I20260812 01:11:08.710987 21239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:08.711210 21246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:08.717572 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:08.673620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.20.151.62:40153)
0812 01:11:08.674772 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:08.674789 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:08.676231 (+ 1442us) server_negotiation.cc:407] Connection header received
0812 01:11:08.687111 (+ 10880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:08.687140 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:08.687650 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:08.688110 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:08.691164 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:08.693425 (+ 2261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:08.695310 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:08.696538 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:08.696729 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:08.703538 (+ 6809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:08.703550 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:08.703574 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:08.703639 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:08.708364 (+ 4725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:08.709739 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:08.709749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:08.709758 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:08.710037 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:08.710752 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:08.710758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:08.710761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:08.711824 (+ 1063us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:08.715111 (+ 3287us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:08.715444 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 01:11:08.715957 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":559,"threads_started":1}
I20260812 01:11:08.728113 21246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:08.744449 21246 catalog_manager.cc:1347] Generated new cluster ID: 00e00b99456c44bfb9576c19a709903c
I20260812 01:11:08.744640 21246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:08.755769 21246 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:08.757241 21246 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:08.764241 21246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87aca3f83d7d4011b9cb10ad0cc54c1a: Generated new TSK 0
I20260812 01:11:08.765344 21246 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:11:08.826534 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40153
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:09.205360 21270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:09.205693 21270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:09.205838 21270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:09.213850 21270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:09.213984 21270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:09.214139 21270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:11:09.224749 21270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:09.226136 21270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:09.227612 21270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:09.241119 21283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:09.241127 21285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:09.241338 21288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:09.242698 21270 server_base.cc:1034] running on GCE node
I20260812 01:11:09.243383 21270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:09.244379 21270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:09.245626 21270 hybrid_clock.cc:648] HybridClock initialized: now 1786497069245562 us; error 36 us; skew 500 ppm
I20260812 01:11:09.246016 21270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:09 dist-test-slave-21b9 krb5kdc[21095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:09.252352 21270 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:11:09.254012 21270 webserver.cc:460] Webserver started at http://127.20.151.1:37531/ using document root <none> and password file <none>
I20260812 01:11:09.254734 21270 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:09.254877 21270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:09.255198 21270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:09.258733 21270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "f42d486d05db41d28af5d5e9135dd555"
format_stamp: "Formatted at 2026-08-12 01:11:09 on dist-test-slave-21b9"
server_key: "a2d0bd34399ccff4517b53aea9991808"
server_key_iv: "886d32f1775db9cf37b05e62441ab2d1"
server_key_version: "encryptionkey@0"
I20260812 01:11:09.259707 21270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "f42d486d05db41d28af5d5e9135dd555"
format_stamp: "Formatted at 2026-08-12 01:11:09 on dist-test-slave-21b9"
server_key: "a2d0bd34399ccff4517b53aea9991808"
server_key_iv: "886d32f1775db9cf37b05e62441ab2d1"
server_key_version: "encryptionkey@0"
I20260812 01:11:09.265215 21270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:11:09.269127 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:09.270646 21270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:11:09.270823 21270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "f42d486d05db41d28af5d5e9135dd555"
format_stamp: "Formatted at 2026-08-12 01:11:09 on dist-test-slave-21b9"
server_key: "a2d0bd34399ccff4517b53aea9991808"
server_key_iv: "886d32f1775db9cf37b05e62441ab2d1"
server_key_version: "encryptionkey@0"
I20260812 01:11:09.271031 21270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:09.286952 21270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:09.291702 21270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:09.292007 21270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:09.293104 21270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:09.294605 21270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:09.294691 21270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:09.294876 21270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:09.294948 21270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:09.332190 21270 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:34279
I20260812 01:11:09.332309 21475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:34279 every 8 connection(s)
I20260812 01:11:09.333608 21270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:11:09.339550 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 21270
I20260812 01:11:09.339785 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:11:09.340410 21084 external_mini_cluster.cc:1428] Setting key 88fa971e13b6e5de7b51798483b33222
Aug 12 01:11:09 dist-test-slave-21b9 krb5kdc[21095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:09.355965 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:09.336440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:33391 (local address 127.20.151.62:40153)
0812 01:11:09.336906 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:09.336911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:09.337885 (+ 974us) server_negotiation.cc:407] Connection header received
0812 01:11:09.339942 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:09.339949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:09.340046 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:09.340143 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:09.342004 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:09.343032 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:09.344788 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:09.345397 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:09.345540 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:09.349460 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:09.349466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:09.349469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:09.349506 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:09.352747 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:09.353542 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:09.353548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:09.353550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:09.353713 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:09.354141 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:09.354146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:09.354150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:09.354496 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:09.354646 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:09.355389 (+ 743us) server_negotiation.cc:299] Negotiation successful
0812 01:11:09.355615 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260812 01:11:09.356909 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:09.336889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40153 (local address 127.20.151.1:33391)
0812 01:11:09.337722 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:09.337762 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:09.339623 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:09.340419 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:09.340428 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:09.340979 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:09.341817 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:09.341844 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:09.343225 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:09.343229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:09.344620 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:09.344635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:09.345601 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:09.345606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:09.345838 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:09.346530 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:09.346555 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:09.349199 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:09.352912 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:09.352920 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:09.352935 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:09.353396 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:09.353815 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:09.353818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:09.353820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:09.354031 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:09.354610 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:09.354616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:09.354968 (+ 352us) client_negotiation.cc:769] Sending connection context
0812 01:11:09.355206 (+ 238us) client_negotiation.cc:240] Negotiation successful
0812 01:11:09.355716 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":359,"threads_started":1}
I20260812 01:11:09.358505 21476 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40153
I20260812 01:11:09.358881 21476 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:09.359884 21476 heartbeater.cc:511] Master 127.20.151.62:40153 requested a full tablet report, sending...
I20260812 01:11:09.362360 21164 ts_manager.cc:194] Registered new tserver with Master: f42d486d05db41d28af5d5e9135dd555 (127.20.151.1:34279)
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
I20260812 01:11:09.364082 21164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:33391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:11:09.390422 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40153
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:09.760308 21484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:09.760639 21484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:09.760790 21484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:09.768695 21484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:09.768834 21484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:09.768992 21484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:11:09.779603 21484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:09.780937 21484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:09.782279 21484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:09.796000 21489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:09.796123 21491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:09.796732 21495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:09.797546 21484 server_base.cc:1034] running on GCE node
I20260812 01:11:09.798195 21484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:09.799208 21484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:09.800509 21484 hybrid_clock.cc:648] HybridClock initialized: now 1786497069800383 us; error 59 us; skew 500 ppm
I20260812 01:11:09.800833 21484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:09 dist-test-slave-21b9 krb5kdc[21095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:09.804157 21484 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:11:09.805739 21484 webserver.cc:460] Webserver started at http://127.20.151.2:41483/ using document root <none> and password file <none>
I20260812 01:11:09.806474 21484 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:09.806603 21484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:09.806916 21484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:09.810434 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "6854179524b549f28bfb3aa5361ba544"
format_stamp: "Formatted at 2026-08-12 01:11:09 on dist-test-slave-21b9"
server_key: "87cd30dd5cce642c593027f6dff66872"
server_key_iv: "b8af474d28e1a1b5133cb5d2505abf18"
server_key_version: "encryptionkey@0"
I20260812 01:11:09.811242 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "6854179524b549f28bfb3aa5361ba544"
format_stamp: "Formatted at 2026-08-12 01:11:09 on dist-test-slave-21b9"
server_key: "87cd30dd5cce642c593027f6dff66872"
server_key_iv: "b8af474d28e1a1b5133cb5d2505abf18"
server_key_version: "encryptionkey@0"
I20260812 01:11:09.816730 21484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:11:09.820290 21509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:09.821595 21484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:09.821754 21484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "6854179524b549f28bfb3aa5361ba544"
format_stamp: "Formatted at 2026-08-12 01:11:09 on dist-test-slave-21b9"
server_key: "87cd30dd5cce642c593027f6dff66872"
server_key_iv: "b8af474d28e1a1b5133cb5d2505abf18"
server_key_version: "encryptionkey@0"
I20260812 01:11:09.821942 21484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:09.838001 21484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:09.842427 21484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:09.842681 21484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:09.843668 21484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:09.845145 21484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:09.845218 21484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:09.845315 21484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:09.845359 21484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:09.880528 21484 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:35123
I20260812 01:11:09.880637 21701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:35123 every 8 connection(s)
I20260812 01:11:09.881806 21484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:11:09.891834 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 21484
I20260812 01:11:09.892031 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:11:09.892624 21084 external_mini_cluster.cc:1428] Setting key ade71af776e44e06731a0ddcf5dc4258
Aug 12 01:11:09 dist-test-slave-21b9 krb5kdc[21095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497069, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:09.903451 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:09.884426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:53705 (local address 127.20.151.62:40153)
0812 01:11:09.884865 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:09.884870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:09.885915 (+ 1045us) server_negotiation.cc:407] Connection header received
0812 01:11:09.887934 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:09.887940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:09.888025 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:09.888112 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:09.889650 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:09.890489 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:09.892201 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:09.892841 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:09.892980 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:09.897147 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:09.897152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:09.897156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:09.897189 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:09.899809 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:09.900729 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:09.900732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:09.900734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:09.900867 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:09.901389 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:09.901393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:09.901396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:09.901731 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:09.901869 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:09.902797 (+ 928us) server_negotiation.cc:299] Negotiation successful
0812 01:11:09.903036 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
I20260812 01:11:09.904829 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:09.884981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40153 (local address 127.20.151.2:53705)
0812 01:11:09.885745 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:09.885783 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:09.887651 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:09.888313 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:09.888321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:09.888893 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:09.889485 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:09.889503 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:09.890704 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:09.890708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:09.892050 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:09.892064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:09.893069 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:09.893075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:09.893290 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:09.893978 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:09.894005 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:09.896880 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:09.899957 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:09.899966 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:09.899985 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:09.900617 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:09.900969 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:09.900973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:09.900977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:09.901257 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:09.901863 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:09.901871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:09.902348 (+ 477us) client_negotiation.cc:769] Sending connection context
0812 01:11:09.902700 (+ 352us) client_negotiation.cc:240] Negotiation successful
0812 01:11:09.903338 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":275,"threads_started":1}
I20260812 01:11:09.906368 21702 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40153
I20260812 01:11:09.906692 21702 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:09.907590 21702 heartbeater.cc:511] Master 127.20.151.62:40153 requested a full tablet report, sending...
I20260812 01:11:09.909348 21164 ts_manager.cc:194] Registered new tserver with Master: 6854179524b549f28bfb3aa5361ba544 (127.20.151.2:35123)
I20260812 01:11:09.910332 21164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:53705
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:11:09.942031 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40153
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:11:10.367509 21476 heartbeater.cc:503] Master 127.20.151.62:40153 was elected leader, sending a full tablet report...
W20260812 01:11:10.389711 21710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:10.390059 21710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:10.390233 21710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:10.399142 21710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:10.399341 21710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:10.399556 21710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:11:10.412554 21710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:10.413937 21710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:10.415537 21710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:10.432103 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:10.432469 21722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:10.432626 21718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:10.434125 21710 server_base.cc:1034] running on GCE node
I20260812 01:11:10.434916 21710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:10.436023 21710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:10.437292 21710 hybrid_clock.cc:648] HybridClock initialized: now 1786497070437233 us; error 36 us; skew 500 ppm
I20260812 01:11:10.437801 21710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:10 dist-test-slave-21b9 krb5kdc[21095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497070, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:10.442292 21710 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:11:10.444173 21710 webserver.cc:460] Webserver started at http://127.20.151.3:34989/ using document root <none> and password file <none>
I20260812 01:11:10.445122 21710 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:10.445343 21710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:10.445756 21710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:10.449435 21710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "4f4fd9e3faa644f38102d49c997a0dbe"
format_stamp: "Formatted at 2026-08-12 01:11:10 on dist-test-slave-21b9"
server_key: "8a0ee7db3d10ec6fdc2e0530a003eb34"
server_key_iv: "1e5f91ee0ef67dd047d0320b44481b0b"
server_key_version: "encryptionkey@0"
I20260812 01:11:10.450440 21710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "4f4fd9e3faa644f38102d49c997a0dbe"
format_stamp: "Formatted at 2026-08-12 01:11:10 on dist-test-slave-21b9"
server_key: "8a0ee7db3d10ec6fdc2e0530a003eb34"
server_key_iv: "1e5f91ee0ef67dd047d0320b44481b0b"
server_key_version: "encryptionkey@0"
I20260812 01:11:10.457248 21710 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260812 01:11:10.462150 21732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:10.464044 21710 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 01:11:10.464231 21710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "4f4fd9e3faa644f38102d49c997a0dbe"
format_stamp: "Formatted at 2026-08-12 01:11:10 on dist-test-slave-21b9"
server_key: "8a0ee7db3d10ec6fdc2e0530a003eb34"
server_key_iv: "1e5f91ee0ef67dd047d0320b44481b0b"
server_key_version: "encryptionkey@0"
I20260812 01:11:10.464447 21710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:10.506567 21710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:10.511821 21710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:10.512239 21710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:10.513602 21710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:10.515467 21710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:10.515679 21710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:10.515823 21710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:10.515950 21710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:10.574509 21710 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:44155
I20260812 01:11:10.574694 21931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:44155 every 8 connection(s)
I20260812 01:11:10.576193 21710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:11:10.585902 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 21710
I20260812 01:11:10.586166 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:11:10.586798 21084 external_mini_cluster.cc:1428] Setting key a024cdf1173ac645f6042f1a8a29c11e
Aug 12 01:11:10 dist-test-slave-21b9 krb5kdc[21095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497070, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:10.600308 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:10.579071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:50535 (local address 127.20.151.62:40153)
0812 01:11:10.579790 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:10.579797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:10.581007 (+ 1210us) server_negotiation.cc:407] Connection header received
0812 01:11:10.583304 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:10.583311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:10.583406 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:10.583511 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:10.585498 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.586779 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.589035 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.589829 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.589930 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:10.594357 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:10.594362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:10.594366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:10.594401 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:10.597344 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:10.598067 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:10.598070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:10.598072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:10.598187 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:10.598594 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:10.598597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:10.598598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:10.598862 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:10.598971 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:10.599713 (+ 742us) server_negotiation.cc:299] Negotiation successful
0812 01:11:10.599888 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":403,"threads_started":1}
I20260812 01:11:10.601193 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:10.579749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40153 (local address 127.20.151.3:50535)
0812 01:11:10.580822 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:10.580872 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:10.582994 (+ 2122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:10.583759 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:10.583767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:10.584420 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:10.585268 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:10.585311 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.586980 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.586991 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:10.588866 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:10.588879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.590146 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.590156 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:10.590459 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:10.591431 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:10.591457 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:10.594133 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:10.597481 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:10.597489 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:10.597502 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:10.597968 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:10.598263 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:10.598266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:10.598268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:10.598510 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:10.598959 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:10.598965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:10.599353 (+ 388us) client_negotiation.cc:769] Sending connection context
0812 01:11:10.599607 (+ 254us) client_negotiation.cc:240] Negotiation successful
0812 01:11:10.600019 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":502,"threads_started":1}
I20260812 01:11:10.603032 21932 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40153
I20260812 01:11:10.603400 21932 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:10.604386 21932 heartbeater.cc:511] Master 127.20.151.62:40153 requested a full tablet report, sending...
I20260812 01:11:10.606052 21163 ts_manager.cc:194] Registered new tserver with Master: 4f4fd9e3faa644f38102d49c997a0dbe (127.20.151.3:44155)
I20260812 01:11:10.607224 21163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:50535
I20260812 01:11:10.616597 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 01:11:10 dist-test-slave-21b9 krb5kdc[21095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 01:11:10 dist-test-slave-21b9 krb5kdc[21095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:10.675482 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:10.655296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.20.151.1:34279)
0812 01:11:10.656110 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:10.656118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:10.656149 (+ 31us) server_negotiation.cc:407] Connection header received
0812 01:11:10.656261 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:10.656268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:10.656492 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:10.656627 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:10.658947 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.660635 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.662314 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.663301 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.663428 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:10.668439 (+ 5011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:10.668464 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:10.668480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:10.668525 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:10.672249 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:10.672942 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:10.672950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:10.672952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:10.673136 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:10.673550 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:10.673554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:10.673556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:10.673922 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:10.674237 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:10.674570 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 01:11:10.674896 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":406,"threads_started":1}
W20260812 01:11:10.676939 21346 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33040
Aug 12 01:11:10 dist-test-slave-21b9 krb5kdc[21095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 01:11:10 dist-test-slave-21b9 krb5kdc[21095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:10.730230 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:10.713928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.20.151.1:34279)
0812 01:11:10.714733 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:10.714738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:10.714759 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:11:10.714822 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:10.714827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:10.714902 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:10.715011 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:10.717128 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.717911 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.719228 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.719875 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.719999 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:10.724194 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:10.724199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:10.724203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:10.724239 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:10.727599 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:10.728235 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:10.728239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:10.728241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:10.728391 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:10.728832 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:10.728837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:10.728839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:10.729202 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:10.729398 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:10.729675 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 01:11:10.729855 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666}
I20260812 01:11:10.731832 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 21270
I20260812 01:11:10.746977 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:10.732899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.20.151.1:34279)
0812 01:11:10.733118 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:10.733124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:10.733403 (+ 279us) server_negotiation.cc:407] Connection header received
0812 01:11:10.733520 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:10.733523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:10.733606 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:10.733679 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:10.735216 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.735973 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.737586 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.737994 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.738087 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:10.740816 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:10.740821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:10.740824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:10.740855 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:10.744230 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:10.744887 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:10.744891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:10.744894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:10.745061 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:10.745422 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:10.745425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:10.745427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:10.745700 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:10.746046 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:10.746249 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 01:11:10.746447 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 01:11:10.748056 21467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:10.833981 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 21270
I20260812 01:11:10.861152 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 21484
Aug 12 01:11:10 dist-test-slave-21b9 krb5kdc[21095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:11:10.877449 21959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:10.862093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32784 (local address 127.20.151.2:35123)
0812 01:11:10.862761 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:10.862768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:10.862795 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:11:10.862868 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:10.862874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:10.863094 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:10.863229 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:10.864711 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.865771 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.867035 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.867675 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.867780 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:10.871956 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:10.871969 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:10.871984 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:10.872020 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:10.874651 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:10.875397 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:10.875409 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:10.875413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:10.875577 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:10.875948 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:10.875951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:10.875953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:10.876354 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:10.876617 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:10.876925 (+ 308us) server_negotiation.cc:299] Negotiation successful
0812 01:11:10.877155 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":387,"threads_started":1}
I20260812 01:11:10.878746 21693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:10.959873 21702 heartbeater.cc:503] Master 127.20.151.62:40153 was elected leader, sending a full tablet report...
I20260812 01:11:10.961480 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 21484
I20260812 01:11:10.988195 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 21710
Aug 12 01:11:10 dist-test-slave-21b9 krb5kdc[21095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:11:11.004122 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:10.989088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.20.151.3:44155)
0812 01:11:10.989654 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:10.989662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:10.989693 (+ 31us) server_negotiation.cc:407] Connection header received
0812 01:11:10.989755 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:10.989761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:10.989967 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:10.990119 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:10.991440 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.992449 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.993733 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:10.994293 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:10.994407 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:10.998622 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:10.998635 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:10.998650 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:10.998690 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:11.001345 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:11.001964 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:11.001970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:11.001973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:11.002103 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:11.002370 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:11.002372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:11.002374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:11.002753 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:11.003043 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:11.003429 (+ 386us) server_negotiation.cc:299] Negotiation successful
0812 01:11:11.003773 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":257,"threads_started":1}
I20260812 01:11:11.005446 21926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:11.088852 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 21710
I20260812 01:11:11.110523 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 21113
I20260812 01:11:11.111799 21223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:11.219208 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 21113
2026-08-12T01:11:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4591 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:11:11 dist-test-slave-21b9 krb5kdc[21970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:11:11 dist-test-slave-21b9 krb5kdc[21970](info): set up 2 sockets
Aug 12 01:11:11 dist-test-slave-21b9 krb5kdc[21970](info): commencing operation
krb5kdc: starting...
W20260812 01:11:12.465513 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:11:12 dist-test-slave-21b9 krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:11:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:11:12.680505 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:40727
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:40727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:13.016330 21990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:13.016680 21990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:13.016834 21990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:13.026129 21990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:13.026245 21990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:13.026328 21990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:13.026407 21990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:13.026468 21990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:11:13.036729 21990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:40727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:40727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:13.038094 21990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:13.039480 21990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:13.051246 22000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:13.051316 22001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:13.051321 22009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:13.052054 21990 server_base.cc:1034] running on GCE node
I20260812 01:11:13.052855 21990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:13.054368 21990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:13.055588 21990 hybrid_clock.cc:648] HybridClock initialized: now 1786497073055528 us; error 50 us; skew 500 ppm
I20260812 01:11:13.055933 21990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:13 dist-test-slave-21b9 krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497073, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:13.059059 21990 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:11:13.060721 21990 webserver.cc:460] Webserver started at http://127.20.151.62:33323/ using document root <none> and password file <none>
I20260812 01:11:13.061420 21990 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:13.061525 21990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:13.061806 21990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:13.064879 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "0126116655bc49f8b125287e5576dcc7"
format_stamp: "Formatted at 2026-08-12 01:11:13 on dist-test-slave-21b9"
server_key: "709c3683c7e130b8d9698d03b1b2247a"
server_key_iv: "c84564c681e11de7b7232bed6b54fa26"
server_key_version: "encryptionkey@0"
I20260812 01:11:13.065604 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "0126116655bc49f8b125287e5576dcc7"
format_stamp: "Formatted at 2026-08-12 01:11:13 on dist-test-slave-21b9"
server_key: "709c3683c7e130b8d9698d03b1b2247a"
server_key_iv: "c84564c681e11de7b7232bed6b54fa26"
server_key_version: "encryptionkey@0"
I20260812 01:11:13.070832 21990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260812 01:11:13.074349 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:13.075730 21990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260812 01:11:13.075896 21990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "0126116655bc49f8b125287e5576dcc7"
format_stamp: "Formatted at 2026-08-12 01:11:13 on dist-test-slave-21b9"
server_key: "709c3683c7e130b8d9698d03b1b2247a"
server_key_iv: "c84564c681e11de7b7232bed6b54fa26"
server_key_version: "encryptionkey@0"
I20260812 01:11:13.076093 21990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:13.090544 21990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:13.094954 21990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:13.095249 21990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:13.113602 21990 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:40727
I20260812 01:11:13.113619 22109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:40727 every 8 connection(s)
I20260812 01:11:13.115049 21990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:13.118901 22111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:13.122473 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 21990
I20260812 01:11:13.122730 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:13.123538 21084 external_mini_cluster.cc:1428] Setting key 5ab61ca9edcb1a92f343a7299b980e50
I20260812 01:11:13.132515 22111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7: Bootstrap starting.
I20260812 01:11:13.136382 22111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:13.137742 22111 log.cc:824] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7: Log is configured to *not* fsync() on all Append() calls
Aug 12 01:11:13 dist-test-slave-21b9 krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:13.141059 22111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7: No bootstrap required, opened a new log
I20260812 01:11:13.147696 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:13.125746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.20.151.62:40727)
0812 01:11:13.126542 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:13.126553 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:13.126595 (+ 42us) server_negotiation.cc:407] Connection header received
0812 01:11:13.128348 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:13.128373 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:13.128799 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:13.129161 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:13.130995 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:13.132370 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:13.134068 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:13.134813 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:13.134962 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:13.139958 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:13.139970 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:13.139993 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:13.140038 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:13.143191 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:13.144046 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:13.144053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:13.144060 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:13.144258 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:13.144646 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:13.144649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:13.144651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:13.145211 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:13.145439 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:13.146055 (+ 616us) server_negotiation.cc:299] Negotiation successful
0812 01:11:13.146524 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":366,"threads_started":1}
I20260812 01:11:13.149770 22111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0126116655bc49f8b125287e5576dcc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40727 } }
I20260812 01:11:13.150142 22111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:13.150261 22111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0126116655bc49f8b125287e5576dcc7, State: Initialized, Role: FOLLOWER
I20260812 01:11:13.150857 22111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0126116655bc49f8b125287e5576dcc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40727 } }
I20260812 01:11:13.151074 22111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:13.151201 22111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:13.151384 22111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:13.153558 22111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0126116655bc49f8b125287e5576dcc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40727 } }
I20260812 01:11:13.154085 22111 leader_election.cc:302] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0126116655bc49f8b125287e5576dcc7; no voters:
I20260812 01:11:13.154789 22111 leader_election.cc:288] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:13.154927 22122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:13.155654 22122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [term 1 LEADER]: Becoming Leader. State: Replica: 0126116655bc49f8b125287e5576dcc7, State: Running, Role: LEADER
I20260812 01:11:13.156194 22122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0126116655bc49f8b125287e5576dcc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40727 } }
I20260812 01:11:13.156898 22111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:13.159047 22123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0126116655bc49f8b125287e5576dcc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0126116655bc49f8b125287e5576dcc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40727 } } }
I20260812 01:11:13.159801 22123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:13.159452 22125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0126116655bc49f8b125287e5576dcc7. Latest consensus state: current_term: 1 leader_uuid: "0126116655bc49f8b125287e5576dcc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0126116655bc49f8b125287e5576dcc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40727 } } }
I20260812 01:11:13.160202 22125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:13.161927 22136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:13.167191 22136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:13.176952 22136 catalog_manager.cc:1347] Generated new cluster ID: b5ef79e16551418d9de5a32ff50ea096
I20260812 01:11:13.177075 22136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:13.204946 22136 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:13.206398 22136 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:13.214429 22136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0126116655bc49f8b125287e5576dcc7: Generated new TSK 0
I20260812 01:11:13.215338 22136 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:11:13.268546 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40727
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:13.606674 22158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:13.606969 22158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:13.607115 22158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:13.614796 22158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:13.614923 22158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:13.615110 22158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:11:13.625178 22158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:13.626585 22158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:13.627978 22158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:13.641081 22167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:13.641115 22166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:13.641191 22170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:13.641970 22158 server_base.cc:1034] running on GCE node
I20260812 01:11:13.642613 22158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:13.643652 22158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:13.644860 22158 hybrid_clock.cc:648] HybridClock initialized: now 1786497073644801 us; error 54 us; skew 500 ppm
I20260812 01:11:13.645181 22158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:13 dist-test-slave-21b9 krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497073, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:13.648386 22158 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:11:13.649837 22158 webserver.cc:460] Webserver started at http://127.20.151.1:44299/ using document root <none> and password file <none>
I20260812 01:11:13.650530 22158 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:13.650653 22158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:13.650957 22158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:13.653986 22158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "a3bcd37fe92c48a2b60e84f162b3ba99"
format_stamp: "Formatted at 2026-08-12 01:11:13 on dist-test-slave-21b9"
server_key: "1763eb4aa392ca67a9d78c17e333a96f"
server_key_iv: "4f979f6f79e855944c8e68d6ab8358d3"
server_key_version: "encryptionkey@0"
I20260812 01:11:13.654805 22158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "a3bcd37fe92c48a2b60e84f162b3ba99"
format_stamp: "Formatted at 2026-08-12 01:11:13 on dist-test-slave-21b9"
server_key: "1763eb4aa392ca67a9d78c17e333a96f"
server_key_iv: "4f979f6f79e855944c8e68d6ab8358d3"
server_key_version: "encryptionkey@0"
I20260812 01:11:13.660458 22158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 01:11:13.664178 22195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:13.665541 22158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:13.665714 22158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "a3bcd37fe92c48a2b60e84f162b3ba99"
format_stamp: "Formatted at 2026-08-12 01:11:13 on dist-test-slave-21b9"
server_key: "1763eb4aa392ca67a9d78c17e333a96f"
server_key_iv: "4f979f6f79e855944c8e68d6ab8358d3"
server_key_version: "encryptionkey@0"
I20260812 01:11:13.665931 22158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:13.685546 22158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:13.690070 22158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:13.690356 22158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:13.691426 22158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:13.692890 22158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:13.692973 22158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:13.693113 22158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:13.693172 22158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:13.725457 22158 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:33179
I20260812 01:11:13.725504 22404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:33179 every 8 connection(s)
I20260812 01:11:13.726745 22158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:11:13.729833 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 22158
I20260812 01:11:13.730064 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:11:13.730898 21084 external_mini_cluster.cc:1428] Setting key 3d49c16089b8e04d83fda63dc9198345
Aug 12 01:11:13 dist-test-slave-21b9 krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497073, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:13.747833 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:13.729285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:50131 (local address 127.20.151.62:40727)
0812 01:11:13.729772 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:13.729777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:13.730835 (+ 1058us) server_negotiation.cc:407] Connection header received
0812 01:11:13.732949 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:13.732955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:13.733061 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:13.733169 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:13.734802 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:13.735873 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:13.737455 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:13.737980 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:13.738115 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:13.742035 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:13.742041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:13.742045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:13.742079 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:13.744628 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:13.745383 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:13.745389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:13.745393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:13.745522 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:13.745961 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:13.745965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:13.745967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:13.746249 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:13.746404 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:13.747180 (+ 776us) server_negotiation.cc:299] Negotiation successful
0812 01:11:13.747470 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":247,"threads_started":1}
I20260812 01:11:13.748884 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:13.729875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40727 (local address 127.20.151.1:50131)
0812 01:11:13.730671 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:13.730710 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:13.732624 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:13.733391 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:13.733399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:13.733985 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:13.734615 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:13.734635 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:13.736032 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:13.736036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:13.737307 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:13.737320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:13.738190 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:13.738194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:13.738396 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:13.739147 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:13.739177 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:13.741787 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:13.744787 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:13.744798 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:13.744816 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:13.745264 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:13.745632 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:13.745637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:13.745641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:13.745851 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:13.746383 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:13.746391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:13.746780 (+ 389us) client_negotiation.cc:769] Sending connection context
0812 01:11:13.747046 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 01:11:13.747680 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":315,"threads_started":1}
I20260812 01:11:13.750763 22409 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40727
I20260812 01:11:13.751081 22409 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
I20260812 01:11:13.751919 22409 heartbeater.cc:511] Master 127.20.151.62:40727 requested a full tablet report, sending...
I20260812 01:11:13.754172 22054 ts_manager.cc:194] Registered new tserver with Master: a3bcd37fe92c48a2b60e84f162b3ba99 (127.20.151.1:33179)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
I20260812 01:11:13.755820 22054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:50131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:11:13.782653 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40727
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:14.136888 22423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:14.137184 22423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:14.137329 22423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:14.144704 22423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:14.144838 22423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:14.145017 22423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:11:14.155020 22423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:14.156378 22423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:14.157760 22423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:14.171200 22433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:14.171200 22435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:14.171228 22431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:14.172009 22423 server_base.cc:1034] running on GCE node
I20260812 01:11:14.172642 22423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:14.173573 22423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:14.174798 22423 hybrid_clock.cc:648] HybridClock initialized: now 1786497074174727 us; error 48 us; skew 500 ppm
I20260812 01:11:14.175092 22423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:14 dist-test-slave-21b9 krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497074, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:14.178148 22423 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:11:14.179600 22423 webserver.cc:460] Webserver started at http://127.20.151.2:46791/ using document root <none> and password file <none>
I20260812 01:11:14.180310 22423 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:14.180449 22423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:14.180755 22423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:14.183760 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "ef98a25f2da14642b9744bfedfcdc030"
format_stamp: "Formatted at 2026-08-12 01:11:14 on dist-test-slave-21b9"
server_key: "e2846bb25b670b6fb351ddfbeea2673f"
server_key_iv: "d024a956515af26b60771795b419a15a"
server_key_version: "encryptionkey@0"
I20260812 01:11:14.184520 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "ef98a25f2da14642b9744bfedfcdc030"
format_stamp: "Formatted at 2026-08-12 01:11:14 on dist-test-slave-21b9"
server_key: "e2846bb25b670b6fb351ddfbeea2673f"
server_key_iv: "d024a956515af26b60771795b419a15a"
server_key_version: "encryptionkey@0"
I20260812 01:11:14.189787 22423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:11:14.193356 22446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:14.194674 22423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:11:14.194839 22423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "ef98a25f2da14642b9744bfedfcdc030"
format_stamp: "Formatted at 2026-08-12 01:11:14 on dist-test-slave-21b9"
server_key: "e2846bb25b670b6fb351ddfbeea2673f"
server_key_iv: "d024a956515af26b60771795b419a15a"
server_key_version: "encryptionkey@0"
I20260812 01:11:14.195040 22423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:14.210551 22423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:14.214807 22423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:14.215111 22423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:14.216182 22423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:14.217528 22423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:14.217604 22423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:14.217701 22423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:14.217757 22423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:14.249771 22423 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:39269
I20260812 01:11:14.249826 22631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:39269 every 8 connection(s)
I20260812 01:11:14.251092 22423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:11:14.252033 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 22423
I20260812 01:11:14.252245 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:11:14.252852 21084 external_mini_cluster.cc:1428] Setting key c8ae4198714d2145997bf7d1c4884d15
Aug 12 01:11:14 dist-test-slave-21b9 krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497074, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
I20260812 01:11:14.278156 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:14.254188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:56031 (local address 127.20.151.62:40727)
0812 01:11:14.254663 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:14.254669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:14.255811 (+ 1142us) server_negotiation.cc:407] Connection header received
0812 01:11:14.258390 (+ 2579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:14.258394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:14.258459 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:14.258554 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:14.260591 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.261398 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.263445 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.264078 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.264200 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:14.272082 (+ 7882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:14.272088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:14.272091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:14.272126 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:14.274799 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:14.275646 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:14.275651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:14.275653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:14.275788 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:14.276257 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:14.276260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:14.276262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:14.276542 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:14.276680 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:14.277557 (+ 877us) server_negotiation.cc:299] Negotiation successful
0812 01:11:14.277791 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":246,"threads_started":1}
I20260812 01:11:14.279188 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:14.254805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40727 (local address 127.20.151.2:56031)
0812 01:11:14.255638 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:14.255686 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:14.258117 (+ 2431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:14.258820 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:14.258830 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:14.259578 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:14.260419 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:14.260443 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.261553 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.261558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:14.263253 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:14.263300 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.264297 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.264302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:14.264521 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:14.268832 (+ 4311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:14.268857 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:14.271799 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:14.274947 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:14.274958 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:14.274976 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:14.275514 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:14.275900 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:14.275906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:14.275910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:14.276136 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:14.276673 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:14.276682 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:14.277117 (+ 435us) client_negotiation.cc:769] Sending connection context
0812 01:11:14.277429 (+ 312us) client_negotiation.cc:240] Negotiation successful
0812 01:11:14.277990 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":909,"threads_started":1}
I20260812 01:11:14.280798 22633 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40727
I20260812 01:11:14.281148 22633 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:14.282047 22633 heartbeater.cc:511] Master 127.20.151.62:40727 requested a full tablet report, sending...
I20260812 01:11:14.283769 22054 ts_manager.cc:194] Registered new tserver with Master: ef98a25f2da14642b9744bfedfcdc030 (127.20.151.2:39269)
I20260812 01:11:14.284657 22054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:56031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:11:14.302393 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40727
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:14.635452 22642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:14.635737 22642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:14.635847 22642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:14.643246 22642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:14.643399 22642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:14.643535 22642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:11:14.653656 22642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:14.654922 22642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:14.656322 22642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:14.669517 22667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:14.669561 22659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:14.669601 22663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:14.670591 22642 server_base.cc:1034] running on GCE node
I20260812 01:11:14.671310 22642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:14.672290 22642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:14.673501 22642 hybrid_clock.cc:648] HybridClock initialized: now 1786497074673448 us; error 46 us; skew 500 ppm
I20260812 01:11:14.673839 22642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:14 dist-test-slave-21b9 krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497074, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:14.677043 22642 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:11:14.678511 22642 webserver.cc:460] Webserver started at http://127.20.151.3:39891/ using document root <none> and password file <none>
I20260812 01:11:14.679244 22642 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:14.679404 22642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:14.679724 22642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:14.682714 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "39f9c934786f4ac9be493984221c5dea"
format_stamp: "Formatted at 2026-08-12 01:11:14 on dist-test-slave-21b9"
server_key: "88003649d4930b7f9730c4641a5ded7f"
server_key_iv: "6746630b8377a308948b9289ec82c646"
server_key_version: "encryptionkey@0"
I20260812 01:11:14.683531 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "39f9c934786f4ac9be493984221c5dea"
format_stamp: "Formatted at 2026-08-12 01:11:14 on dist-test-slave-21b9"
server_key: "88003649d4930b7f9730c4641a5ded7f"
server_key_iv: "6746630b8377a308948b9289ec82c646"
server_key_version: "encryptionkey@0"
I20260812 01:11:14.688612 22642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 01:11:14.692155 22680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:14.693485 22642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:14.693662 22642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "39f9c934786f4ac9be493984221c5dea"
format_stamp: "Formatted at 2026-08-12 01:11:14 on dist-test-slave-21b9"
server_key: "88003649d4930b7f9730c4641a5ded7f"
server_key_iv: "6746630b8377a308948b9289ec82c646"
server_key_version: "encryptionkey@0"
I20260812 01:11:14.693861 22642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:14.709021 22642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:14.713193 22642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:14.713511 22642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:14.714700 22642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:14.716217 22642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:14.716297 22642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:14.716414 22642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:14.716470 22642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:14.756166 22642 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:38987
I20260812 01:11:14.756439 22865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:38987 every 8 connection(s)
I20260812 01:11:14.757642 22642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:11:14.759107 22409 heartbeater.cc:503] Master 127.20.151.62:40727 was elected leader, sending a full tablet report...
I20260812 01:11:14.763463 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 22642
I20260812 01:11:14.763660 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:11:14.764144 21084 external_mini_cluster.cc:1428] Setting key a22a1c63feb92155bd1aee4e3077c755
Aug 12 01:11:14 dist-test-slave-21b9 krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497074, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:14.781458 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:14.760782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:48845 (local address 127.20.151.62:40727)
0812 01:11:14.760951 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:14.760955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:14.762500 (+ 1545us) server_negotiation.cc:407] Connection header received
0812 01:11:14.764886 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:14.764893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:14.764985 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:14.765090 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:14.767258 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.768323 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.770300 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.770701 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.770793 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:14.774832 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:14.774837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:14.774840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:14.774873 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:14.777689 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:14.778562 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:14.778568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:14.778571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:14.778742 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:14.779493 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:14.779499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:14.779502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:14.779916 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:14.780166 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:14.780934 (+ 768us) server_negotiation.cc:299] Negotiation successful
0812 01:11:14.781164 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 01:11:14.782531 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:14.761218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40727 (local address 127.20.151.3:48845)
0812 01:11:14.762335 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:14.762387 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:14.764564 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:14.765367 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:14.765380 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:14.766065 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:14.767066 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:14.767109 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.768468 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.768473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:14.770159 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:14.770174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.770895 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.770901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:14.771189 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:14.771983 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:14.772007 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:14.774630 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:14.777816 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:14.777824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:14.777837 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:14.778435 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:14.778856 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:14.778861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:14.778865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:14.779374 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:14.780120 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:14.780129 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:14.780536 (+ 407us) client_negotiation.cc:769] Sending connection context
0812 01:11:14.780826 (+ 290us) client_negotiation.cc:240] Negotiation successful
0812 01:11:14.781277 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":492,"threads_started":1}
I20260812 01:11:14.784085 22873 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40727
I20260812 01:11:14.784403 22873 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:14.785499 22873 heartbeater.cc:511] Master 127.20.151.62:40727 requested a full tablet report, sending...
I20260812 01:11:14.787102 22054 ts_manager.cc:194] Registered new tserver with Master: 39f9c934786f4ac9be493984221c5dea (127.20.151.3:38987)
I20260812 01:11:14.788022 22054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:48845
I20260812 01:11:14.793175 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:11:14.816866 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:14.801315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.20.151.62:40727)
0812 01:11:14.801593 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:14.801600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:14.802219 (+ 619us) server_negotiation.cc:407] Connection header received
0812 01:11:14.802381 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:14.802385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:14.802465 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:14.802585 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:14.804954 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.805845 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.807347 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.807848 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.807948 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:14.810614 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:14.810619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:14.810622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:14.810656 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:14.813744 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:14.814617 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:14.814622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:14.814624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:14.814774 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:14.815138 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:14.815141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:14.815143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:14.815630 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:14.815852 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:14.816200 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 01:11:14.816378 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260812 01:11:14.823889 22054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:11:14.833307 22054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:11:14.862305 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:14.849367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.20.151.2:39269)
0812 01:11:14.850292 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:14.850301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:14.852557 (+ 2256us) server_negotiation.cc:407] Connection header received
0812 01:11:14.852679 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:14.852687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:14.852908 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:14.853094 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:14.854165 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.855782 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.859170 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.860979 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.861070 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:14.861216 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:14.861734 (+ 518us) server_negotiation.cc:299] Negotiation successful
0812 01:11:14.861901 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":590,"threads_started":1}
I20260812 01:11:14.862687 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:14.848977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:39269 (local address 127.0.0.1:53734)
0812 01:11:14.852268 (+ 3291us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:14.852287 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:14.852427 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:14.853112 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:14.853119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:14.853148 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:14.853793 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:14.853807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.855969 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.855974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:14.858932 (+ 2958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:14.858949 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.861357 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.861361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:14.861508 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:14.861540 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:11:14.861654 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 01:11:14.861870 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3169,"thread_start_us":2192,"threads_started":1}
I20260812 01:11:14.863296 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:14.848365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:38987 (local address 127.0.0.1:41584)
0812 01:11:14.852082 (+ 3717us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:14.852131 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:14.852344 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:14.853649 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:14.853653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:14.853675 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:14.854832 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:14.854845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.856773 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.856778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:14.859543 (+ 2765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:14.859557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.862221 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.862232 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:14.862417 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:14.862454 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 01:11:14.862544 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 01:11:14.862678 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2991,"mutex_wait_us":77,"spinlock_wait_cycles":1408,"thread_start_us":3299,"threads_started":1}
I20260812 01:11:14.864331 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:14.848516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.20.151.3:38987)
0812 01:11:14.849785 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:14.849795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:14.852552 (+ 2757us) server_negotiation.cc:407] Connection header received
0812 01:11:14.852645 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:14.852654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:14.852962 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:14.853149 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:14.854986 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.856634 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.859735 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.861463 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.861592 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:14.861885 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:14.862805 (+ 920us) server_negotiation.cc:299] Negotiation successful
0812 01:11:14.863023 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1077,"thread_start_us":745,"threads_started":1}
I20260812 01:11:14.864547 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:14.848594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:33179 (local address 127.0.0.1:49416)
0812 01:11:14.852154 (+ 3560us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:14.852171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:14.852344 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:14.853435 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:14.853440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:14.853473 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:14.854695 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:14.854712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.857060 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.857065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:14.859943 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:14.859959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.862645 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.862653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:14.863797 (+ 1144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:14.863832 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 01:11:14.863910 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 01:11:14.864031 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3359,"mutex_wait_us":2,"spinlock_wait_cycles":5376,"thread_start_us":2424,"threads_started":1}
I20260812 01:11:14.865397 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:14.849047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.20.151.1:33179)
0812 01:11:14.850239 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:14.850247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:14.852533 (+ 2286us) server_negotiation.cc:407] Connection header received
0812 01:11:14.852645 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:14.852654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:14.852959 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:14.853142 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:14.855127 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.856880 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.860108 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:14.862098 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:14.862239 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:14.864755 (+ 2516us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:14.864834 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 01:11:14.865006 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":788,"threads_started":1}
I20260812 01:11:14.867341 22519 tablet_service.cc:1467] Processing CreateTablet for tablet 3ceb7cd203af406997d7e89c3380c937 (DEFAULT_TABLE table=test-table [id=f1c2d09c7b504d38ac7b56daddffe354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:14.868345 22275 tablet_service.cc:1467] Processing CreateTablet for tablet 3ceb7cd203af406997d7e89c3380c937 (DEFAULT_TABLE table=test-table [id=f1c2d09c7b504d38ac7b56daddffe354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:14.868376 22767 tablet_service.cc:1467] Processing CreateTablet for tablet 3ceb7cd203af406997d7e89c3380c937 (DEFAULT_TABLE table=test-table [id=f1c2d09c7b504d38ac7b56daddffe354]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:14.869125 22519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ceb7cd203af406997d7e89c3380c937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:14.869805 22767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ceb7cd203af406997d7e89c3380c937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:14.870036 22275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ceb7cd203af406997d7e89c3380c937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:14.883762 22910 tablet_bootstrap.cc:492] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99: Bootstrap starting.
I20260812 01:11:14.887759 22910 tablet_bootstrap.cc:654] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:14.888334 22911 tablet_bootstrap.cc:492] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030: Bootstrap starting.
I20260812 01:11:14.889578 22910 log.cc:824] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:14.891518 22914 tablet_bootstrap.cc:492] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea: Bootstrap starting.
I20260812 01:11:14.893110 22910 tablet_bootstrap.cc:492] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99: No bootstrap required, opened a new log
I20260812 01:11:14.893368 22910 ts_tablet_manager.cc:1397] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 01:11:14.893965 22911 tablet_bootstrap.cc:654] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:14.895744 22911 log.cc:824] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:14.895794 22914 tablet_bootstrap.cc:654] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:14.897380 22914 log.cc:824] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:14.899672 22911 tablet_bootstrap.cc:492] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030: No bootstrap required, opened a new log
I20260812 01:11:14.899962 22911 ts_tablet_manager.cc:1397] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20260812 01:11:14.901135 22914 tablet_bootstrap.cc:492] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea: No bootstrap required, opened a new log
I20260812 01:11:14.901433 22914 ts_tablet_manager.cc:1397] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 01:11:14.903924 22910 raft_consensus.cc:348] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f9c934786f4ac9be493984221c5dea" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38987 } } peers { permanent_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33179 } } peers { permanent_uuid: "ef98a25f2da14642b9744bfedfcdc030" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39269 } }
I20260812 01:11:14.904366 22910 raft_consensus.cc:374] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:14.904551 22910 raft_consensus.cc:729] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3bcd37fe92c48a2b60e84f162b3ba99, State: Initialized, Role: FOLLOWER
I20260812 01:11:14.905179 22910 consensus_queue.cc:260] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f9c934786f4ac9be493984221c5dea" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38987 } } peers { permanent_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33179 } } peers { permanent_uuid: "ef98a25f2da14642b9744bfedfcdc030" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39269 } }
I20260812 01:11:14.907322 22910 ts_tablet_manager.cc:1428] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260812 01:11:14.912897 22911 raft_consensus.cc:348] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f9c934786f4ac9be493984221c5dea" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38987 } } peers { permanent_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33179 } } peers { permanent_uuid: "ef98a25f2da14642b9744bfedfcdc030" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39269 } }
I20260812 01:11:14.913460 22911 raft_consensus.cc:374] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:14.913591 22911 raft_consensus.cc:729] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef98a25f2da14642b9744bfedfcdc030, State: Initialized, Role: FOLLOWER
I20260812 01:11:14.914016 22914 raft_consensus.cc:348] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f9c934786f4ac9be493984221c5dea" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38987 } } peers { permanent_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33179 } } peers { permanent_uuid: "ef98a25f2da14642b9744bfedfcdc030" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39269 } }
I20260812 01:11:14.914579 22914 raft_consensus.cc:374] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:14.914412 22911 consensus_queue.cc:260] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f9c934786f4ac9be493984221c5dea" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38987 } } peers { permanent_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33179 } } peers { permanent_uuid: "ef98a25f2da14642b9744bfedfcdc030" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39269 } }
I20260812 01:11:14.914690 22914 raft_consensus.cc:729] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f9c934786f4ac9be493984221c5dea, State: Initialized, Role: FOLLOWER
I20260812 01:11:14.915465 22914 consensus_queue.cc:260] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f9c934786f4ac9be493984221c5dea" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38987 } } peers { permanent_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33179 } } peers { permanent_uuid: "ef98a25f2da14642b9744bfedfcdc030" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39269 } }
I20260812 01:11:14.917004 22873 heartbeater.cc:503] Master 127.20.151.62:40727 was elected leader, sending a full tablet report...
I20260812 01:11:14.917912 22914 ts_tablet_manager.cc:1428] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260812 01:11:14.919260 22633 heartbeater.cc:503] Master 127.20.151.62:40727 was elected leader, sending a full tablet report...
I20260812 01:11:14.919911 22911 ts_tablet_manager.cc:1428] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030: Time spent starting tablet: real 0.020s user 0.017s sys 0.003s
W20260812 01:11:14.979239 22412 tablet.cc:2369] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:14.996938 22921 raft_consensus.cc:482] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:11:14.997292 22921 raft_consensus.cc:504] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f9c934786f4ac9be493984221c5dea" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38987 } } peers { permanent_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33179 } } peers { permanent_uuid: "ef98a25f2da14642b9744bfedfcdc030" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39269 } }
I20260812 01:11:14.999629 22921 leader_election.cc:288] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3bcd37fe92c48a2b60e84f162b3ba99 (127.20.151.1:33179), ef98a25f2da14642b9744bfedfcdc030 (127.20.151.2:39269)
W20260812 01:11:15.003173 22634 tablet.cc:2369] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:15.007177 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:15.000084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:39181 (local address 127.20.151.1:33179)
0812 01:11:15.000341 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:15.000348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:15.000896 (+ 548us) server_negotiation.cc:407] Connection header received
0812 01:11:15.001070 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:15.001076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:15.001162 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:15.001274 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:15.001852 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.002677 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.005121 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.006267 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.006440 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:15.006507 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:15.006797 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 01:11:15.006922 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 01:11:15.007205 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:15.000605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:33179 (local address 127.20.151.3:39181)
0812 01:11:15.000777 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:15.000797 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:15.000931 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:15.001321 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:15.001327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:15.001358 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:15.001682 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:15.001698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.002813 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.002820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:15.004784 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:15.004803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.006434 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.006440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:15.006635 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:15.006672 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 01:11:15.006780 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 01:11:15.006917 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 01:11:15.008277 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ceb7cd203af406997d7e89c3380c937" candidate_uuid: "39f9c934786f4ac9be493984221c5dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99" is_pre_election: true
I20260812 01:11:15.009161 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:15.000461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:40303 (local address 127.20.151.2:39269)
0812 01:11:15.000706 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:15.000711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:15.002070 (+ 1359us) server_negotiation.cc:407] Connection header received
0812 01:11:15.002220 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:15.002226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:15.002288 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:15.002362 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:15.003191 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.004297 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.006870 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.008149 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.008304 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:15.008376 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:15.008810 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 01:11:15.008944 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:11:15.009200 22307 raft_consensus.cc:2454] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f9c934786f4ac9be493984221c5dea in term 0.
I20260812 01:11:15.009563 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:15.000612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:39269 (local address 127.20.151.3:40303)
0812 01:11:15.001971 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:15.001984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:15.002090 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:15.002388 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:15.002393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:15.002416 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:15.002998 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:15.003011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.004482 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.004488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:15.006722 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:15.006735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.008357 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.008363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:15.008610 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:15.008643 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 01:11:15.008749 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 01:11:15.008872 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"mutex_wait_us":54,"thread_start_us":773,"threads_started":1}
W20260812 01:11:15.009747 22875 tablet.cc:2369] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:15.010887 22686 leader_election.cc:302] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f9c934786f4ac9be493984221c5dea, a3bcd37fe92c48a2b60e84f162b3ba99; no voters:
I20260812 01:11:15.011312 22921 raft_consensus.cc:2781] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:11:15.011538 22921 raft_consensus.cc:482] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:11:15.011680 22921 raft_consensus.cc:3037] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:15.011735 22555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ceb7cd203af406997d7e89c3380c937" candidate_uuid: "39f9c934786f4ac9be493984221c5dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef98a25f2da14642b9744bfedfcdc030" is_pre_election: true
I20260812 01:11:15.012329 22555 raft_consensus.cc:2454] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f9c934786f4ac9be493984221c5dea in term 0.
I20260812 01:11:15.014504 22921 raft_consensus.cc:504] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f9c934786f4ac9be493984221c5dea" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38987 } } peers { permanent_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33179 } } peers { permanent_uuid: "ef98a25f2da14642b9744bfedfcdc030" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39269 } }
I20260812 01:11:15.015197 22921 leader_election.cc:288] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [CANDIDATE]: Term 1 election: Requested vote from peers a3bcd37fe92c48a2b60e84f162b3ba99 (127.20.151.1:33179), ef98a25f2da14642b9744bfedfcdc030 (127.20.151.2:39269)
I20260812 01:11:15.015607 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ceb7cd203af406997d7e89c3380c937" candidate_uuid: "39f9c934786f4ac9be493984221c5dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99"
I20260812 01:11:15.015651 22555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ceb7cd203af406997d7e89c3380c937" candidate_uuid: "39f9c934786f4ac9be493984221c5dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef98a25f2da14642b9744bfedfcdc030"
I20260812 01:11:15.015798 22307 raft_consensus.cc:3037] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:15.015889 22555 raft_consensus.cc:3037] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:15.018492 22307 raft_consensus.cc:2454] T 3ceb7cd203af406997d7e89c3380c937 P a3bcd37fe92c48a2b60e84f162b3ba99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f9c934786f4ac9be493984221c5dea in term 1.
I20260812 01:11:15.018594 22555 raft_consensus.cc:2454] T 3ceb7cd203af406997d7e89c3380c937 P ef98a25f2da14642b9744bfedfcdc030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f9c934786f4ac9be493984221c5dea in term 1.
I20260812 01:11:15.018951 22686 leader_election.cc:302] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f9c934786f4ac9be493984221c5dea, a3bcd37fe92c48a2b60e84f162b3ba99; no voters:
I20260812 01:11:15.019400 22921 raft_consensus.cc:2781] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:15.020067 22921 raft_consensus.cc:686] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [term 1 LEADER]: Becoming Leader. State: Replica: 39f9c934786f4ac9be493984221c5dea, State: Running, Role: LEADER
I20260812 01:11:15.020654 22921 consensus_queue.cc:237] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f9c934786f4ac9be493984221c5dea" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38987 } } peers { permanent_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33179 } } peers { permanent_uuid: "ef98a25f2da14642b9744bfedfcdc030" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39269 } }
I20260812 01:11:15.026000 22053 catalog_manager.cc:5495] T 3ceb7cd203af406997d7e89c3380c937 P 39f9c934786f4ac9be493984221c5dea reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f9c934786f4ac9be493984221c5dea (127.20.151.3). New cstate: current_term: 1 leader_uuid: "39f9c934786f4ac9be493984221c5dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f9c934786f4ac9be493984221c5dea" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3bcd37fe92c48a2b60e84f162b3ba99" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef98a25f2da14642b9744bfedfcdc030" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 39269 } health_report { overall_health: UNKNOWN } } }
Aug 12 01:11:15 dist-test-slave-21b9 krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:15.063812 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:15.046192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.20.151.1:33179)
0812 01:11:15.046427 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:15.046434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:15.046956 (+ 522us) server_negotiation.cc:407] Connection header received
0812 01:11:15.047094 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:15.047098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:15.047179 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:15.047291 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:15.049563 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.050601 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.052115 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.052754 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.052861 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:15.057376 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:15.057395 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:15.057413 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:15.057477 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:15.060970 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.061739 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.061747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.061749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.061892 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.062187 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.062190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.062193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.062705 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:15.063008 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:15.063140 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 01:11:15.063344 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 01:11:15 dist-test-slave-21b9 krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497075, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 01:11:15 dist-test-slave-21b9 krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497075, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:15.121727 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:15.105429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49436 (local address 127.20.151.1:33179)
0812 01:11:15.105619 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:15.105626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:15.106013 (+ 387us) server_negotiation.cc:407] Connection header received
0812 01:11:15.106193 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:15.106197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:15.106268 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:15.106354 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:15.108113 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.109106 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.110539 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.111130 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.111241 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:15.115672 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:15.115678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:15.115682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:15.115719 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:15.118988 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.119818 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.119822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.119824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.120011 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.120409 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.120412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.120414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.120720 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:15.120991 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:15.121228 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 01:11:15.121403 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Aug 12 01:11:15 dist-test-slave-21b9 krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 01:11:15 dist-test-slave-21b9 krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:15.174464 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:15.159217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.20.151.1:33179)
0812 01:11:15.159464 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:15.159471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:15.159583 (+ 112us) server_negotiation.cc:407] Connection header received
0812 01:11:15.159773 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:15.159777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:15.159842 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:15.159925 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:15.161603 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.162288 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.163796 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.164230 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.164340 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:15.168762 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:15.168767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:15.168771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:15.168806 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:15.171514 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.172154 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.172158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.172160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.172285 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.172690 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.172694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.172696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.173165 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:15.173440 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:15.173724 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 01:11:15.173997 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260812 01:11:15.176398 22241 tablet_service.cc:2908] Rejecting scan request for tablet 3ceb7cd203af406997d7e89c3380c937: Uninitialized: clean time has not yet been initialized
I20260812 01:11:15.179577 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 22158
I20260812 01:11:15.193317 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:15.180651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.20.151.1:33179)
0812 01:11:15.180812 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:15.180860 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:15.181141 (+ 281us) server_negotiation.cc:407] Connection header received
0812 01:11:15.181269 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:15.181273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:15.181341 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:15.181416 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:15.182501 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.183236 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.184696 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.185228 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.185379 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:15.188227 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:15.188232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:15.188235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:15.188267 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:15.190924 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.191671 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.191675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.191677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.191833 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.192148 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.192151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.192153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.192469 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:15.192775 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:15.192877 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 01:11:15.193051 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 01:11:15.194056 22396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:15.285823 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 22158
I20260812 01:11:15.314489 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 22423
Aug 12 01:11:15 dist-test-slave-21b9 krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:11:15.329138 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:15.315312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53740 (local address 127.20.151.2:39269)
0812 01:11:15.315499 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:15.315504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:15.315523 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:11:15.315616 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:15.315620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:15.315684 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:15.315776 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:15.317043 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.317893 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.319194 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.319727 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.319828 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:15.323709 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:15.323728 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:15.323745 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:15.323783 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:15.326359 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.327054 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.327062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.327064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.327200 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.327540 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.327543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.327545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.328010 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:15.328338 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:15.328523 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 01:11:15.328772 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:11:15.329847 22623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:15.419924 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 22423
I20260812 01:11:15.445369 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 22642
Aug 12 01:11:15 dist-test-slave-21b9 krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:11:15.460507 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:15.446173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.20.151.3:38987)
0812 01:11:15.446629 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:15.446633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:15.446651 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:11:15.446709 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:15.446713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:15.446781 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:15.446878 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:15.448188 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.448899 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.450229 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:15.450678 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:15.450774 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:15.454863 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:15.454881 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:15.454899 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:15.454935 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:15.457880 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.458567 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.458575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.458577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.458721 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:15.459030 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:15.459033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:15.459035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:15.459468 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:15.459716 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:15.459937 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 01:11:15.460135 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":249,"threads_started":1}
I20260812 01:11:15.461202 22861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:15.551813 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 22642
I20260812 01:11:15.577412 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 21990
I20260812 01:11:15.578703 22105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:15.691401 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 21990
2026-08-12T01:11:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4477 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:11:15 dist-test-slave-21b9 krb5kdc[22964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:11:15 dist-test-slave-21b9 krb5kdc[22964](info): set up 2 sockets
Aug 12 01:11:15 dist-test-slave-21b9 krb5kdc[22964](info): commencing operation
krb5kdc: starting...
W20260812 01:11:16.936656 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:11:17 dist-test-slave-21b9 krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:11:17.162736 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:44023
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:44023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:17.507938 22990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:17.508263 22990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:17.508388 22990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:17.516217 22990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:17.516371 22990 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 01:11:17.516440 22990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:17.516515 22990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:17.516578 22990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:17.516654 22990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:11:17.527165 22990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:44023
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:44023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:17.528602 22990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:17.530071 22990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:17.542706 22996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:17.542663 23002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:17.542734 22998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:17.543859 22990 server_base.cc:1034] running on GCE node
I20260812 01:11:17.544914 22990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:17.546463 22990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:17.547694 22990 hybrid_clock.cc:648] HybridClock initialized: now 1786497077547620 us; error 53 us; skew 500 ppm
I20260812 01:11:17.548040 22990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:17 dist-test-slave-21b9 krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497077, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:17.551242 22990 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:11:17.552831 22990 webserver.cc:460] Webserver started at http://127.20.151.62:36307/ using document root <none> and password file <none>
I20260812 01:11:17.553619 22990 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:17.553730 22990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:17.554057 22990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:17.557159 22990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "2418963af3a34a7ca5e092ac2dc43138"
format_stamp: "Formatted at 2026-08-12 01:11:17 on dist-test-slave-21b9"
server_key: "ab7e0ed1774ef504dd107333764f538e"
server_key_iv: "c0d67ef260703b7d8819b3ec1fce5d3e"
server_key_version: "encryptionkey@0"
I20260812 01:11:17.557950 22990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "2418963af3a34a7ca5e092ac2dc43138"
format_stamp: "Formatted at 2026-08-12 01:11:17 on dist-test-slave-21b9"
server_key: "ab7e0ed1774ef504dd107333764f538e"
server_key_iv: "c0d67ef260703b7d8819b3ec1fce5d3e"
server_key_version: "encryptionkey@0"
I20260812 01:11:17.563298 22990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:11:17.566836 23009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:17.568224 22990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:11:17.568394 22990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "2418963af3a34a7ca5e092ac2dc43138"
format_stamp: "Formatted at 2026-08-12 01:11:17 on dist-test-slave-21b9"
server_key: "ab7e0ed1774ef504dd107333764f538e"
server_key_iv: "c0d67ef260703b7d8819b3ec1fce5d3e"
server_key_version: "encryptionkey@0"
I20260812 01:11:17.568614 22990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:17.595875 22990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:17.600034 22990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:17.600312 22990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:17.621819 22990 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:44023
I20260812 01:11:17.621827 23108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:44023 every 8 connection(s)
I20260812 01:11:17.623195 22990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:17.625280 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 22990
I20260812 01:11:17.625520 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:17.626379 21084 external_mini_cluster.cc:1428] Setting key 815424fb5d64df2ef73a59195c6579a4
I20260812 01:11:17.627938 23109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 01:11:17 dist-test-slave-21b9 krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:17.642107 23109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138: Bootstrap starting.
I20260812 01:11:17.646292 23109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:17.647650 23109 log.cc:824] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:17.651080 23109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138: No bootstrap required, opened a new log
I20260812 01:11:17.651324 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:17.628535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.20.151.62:44023)
0812 01:11:17.629348 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:17.629363 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:17.629415 (+ 52us) server_negotiation.cc:407] Connection header received
0812 01:11:17.631489 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:17.631517 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:17.632031 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:17.632493 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:17.633855 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:17.635947 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:17.637183 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:17.638294 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:17.638511 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:17.642725 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:17.642737 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:17.642756 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:17.642811 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:17.646751 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:17.647516 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:17.647524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:17.647534 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:17.647766 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:17.648155 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:17.648165 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:17.648169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:17.648782 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:17.649051 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:17.649523 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 01:11:17.649954 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":333,"threads_started":1}
I20260812 01:11:17.658924 23109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2418963af3a34a7ca5e092ac2dc43138" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44023 } }
I20260812 01:11:17.659343 23109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:17.659437 23109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2418963af3a34a7ca5e092ac2dc43138, State: Initialized, Role: FOLLOWER
I20260812 01:11:17.659989 23109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2418963af3a34a7ca5e092ac2dc43138" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44023 } }
I20260812 01:11:17.660238 23109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:17.660377 23109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:17.660547 23109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:17.662773 23109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2418963af3a34a7ca5e092ac2dc43138" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44023 } }
I20260812 01:11:17.663368 23109 leader_election.cc:302] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2418963af3a34a7ca5e092ac2dc43138; no voters:
I20260812 01:11:17.663959 23109 leader_election.cc:288] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:17.664104 23119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:17.664827 23119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [term 1 LEADER]: Becoming Leader. State: Replica: 2418963af3a34a7ca5e092ac2dc43138, State: Running, Role: LEADER
I20260812 01:11:17.665702 23119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2418963af3a34a7ca5e092ac2dc43138" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44023 } }
I20260812 01:11:17.666214 23109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:17.668654 23120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2418963af3a34a7ca5e092ac2dc43138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2418963af3a34a7ca5e092ac2dc43138" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44023 } } }
I20260812 01:11:17.668687 23122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2418963af3a34a7ca5e092ac2dc43138. Latest consensus state: current_term: 1 leader_uuid: "2418963af3a34a7ca5e092ac2dc43138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2418963af3a34a7ca5e092ac2dc43138" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44023 } } }
I20260812 01:11:17.669096 23122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:17.669771 23120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:17.670712 23142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:17.676401 23142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:17.685575 23142 catalog_manager.cc:1347] Generated new cluster ID: e2b20cc47a2a45eba91b567b49017da0
I20260812 01:11:17.685712 23142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:17.697917 23142 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:17.699916 23142 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:17.711247 23142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138: Generated new TSK 0
I20260812 01:11:17.712430 23142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:11:17.777262 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:44023
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:18.125026 23172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:18.125314 23172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:18.125444 23172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:18.132902 23172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:18.133010 23172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:18.133131 23172 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 01:11:18.133235 23172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:11:18.143036 23172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.151.62:44023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:18.144371 23172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:18.145742 23172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:18.158859 23185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:18.159075 23182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:18.159066 23183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:18.160187 23172 server_base.cc:1034] running on GCE node
I20260812 01:11:18.160816 23172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:18.161703 23172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:18.162904 23172 hybrid_clock.cc:648] HybridClock initialized: now 1786497078162833 us; error 46 us; skew 500 ppm
I20260812 01:11:18.163184 23172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:18 dist-test-slave-21b9 krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497078, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:18.166296 23172 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:11:18.167851 23172 webserver.cc:460] Webserver started at http://127.20.151.1:45123/ using document root <none> and password file <none>
I20260812 01:11:18.168529 23172 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:18.168632 23172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:18.168952 23172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:18.172080 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "3f83fcbdc88e4d679690f2d7ac48881c"
format_stamp: "Formatted at 2026-08-12 01:11:18 on dist-test-slave-21b9"
server_key: "af7a362f245af17c5dee4cd8c545d3ee"
server_key_iv: "5827b4d1da2472a5f962884357a77e8d"
server_key_version: "encryptionkey@0"
I20260812 01:11:18.172896 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "3f83fcbdc88e4d679690f2d7ac48881c"
format_stamp: "Formatted at 2026-08-12 01:11:18 on dist-test-slave-21b9"
server_key: "af7a362f245af17c5dee4cd8c545d3ee"
server_key_iv: "5827b4d1da2472a5f962884357a77e8d"
server_key_version: "encryptionkey@0"
I20260812 01:11:18.178010 23172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 01:11:18.181361 23201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:18.182647 23172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:11:18.182803 23172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "3f83fcbdc88e4d679690f2d7ac48881c"
format_stamp: "Formatted at 2026-08-12 01:11:18 on dist-test-slave-21b9"
server_key: "af7a362f245af17c5dee4cd8c545d3ee"
server_key_iv: "5827b4d1da2472a5f962884357a77e8d"
server_key_version: "encryptionkey@0"
I20260812 01:11:18.183003 23172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:18.195924 23172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:18.199822 23172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:18.200080 23172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:18.202749 23172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:18.202834 23172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:18.203008 23172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:18.203119 23172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 12 01:11:18 dist-test-slave-21b9 krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497078, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:18.226155 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:18.203718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:44023 (local address 127.20.151.1:40437)
0812 01:11:18.204628 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:18.204672 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:18.206599 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:18.207360 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:18.207371 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:18.208077 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:18.208993 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:18.209024 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.210370 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.210375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:18.212367 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:18.212386 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.213385 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.213397 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:18.213590 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:18.214271 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:18.214303 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:18.217355 (+ 3052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:18.220950 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:18.220967 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:18.220986 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:18.221609 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:18.222081 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:18.222086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:18.222090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:18.222409 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:18.223182 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:18.223190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:18.223708 (+ 518us) client_negotiation.cc:769] Sending connection context
0812 01:11:18.224014 (+ 306us) client_negotiation.cc:240] Negotiation successful
0812 01:11:18.224549 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":404,"threads_started":1}
I20260812 01:11:18.227119 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:18.203126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:40437 (local address 127.20.151.62:44023)
0812 01:11:18.203622 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:18.203629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:18.204811 (+ 1182us) server_negotiation.cc:407] Connection header received
0812 01:11:18.206870 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:18.206877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:18.206975 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:18.207068 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:18.209265 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.210153 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.212563 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.213122 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.213235 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:18.217590 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:18.217597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:18.217601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:18.217644 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:18.220816 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:18.221790 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:18.221796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:18.221800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:18.221957 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:18.222576 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:18.222582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:18.222586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:18.222920 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:18.223085 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:18.226677 (+ 3592us) server_negotiation.cc:299] Negotiation successful
0812 01:11:18.226855 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260812 01:11:18.246268 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:18.232662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:44023 (local address 127.20.151.1:55085)
0812 01:11:18.233551 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:18.233574 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:18.233791 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:18.234178 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:18.234182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:18.234618 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:18.234911 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:18.234920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.236172 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.236176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:18.237192 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:18.237204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.238140 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.238147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:18.238335 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:18.239038 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:18.239055 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:18.240096 (+ 1041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:18.242994 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:18.243000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:18.243008 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:18.243592 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:18.244061 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:18.244067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:18.244070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:18.244253 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:18.244941 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:18.244946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:18.245177 (+ 231us) client_negotiation.cc:769] Sending connection context
0812 01:11:18.245341 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 01:11:18.245735 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":479,"threads_started":1}
I20260812 01:11:18.247051 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:18.232609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:55085 (local address 127.20.151.62:44023)
0812 01:11:18.232791 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:18.232797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:18.233662 (+ 865us) server_negotiation.cc:407] Connection header received
0812 01:11:18.233970 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:18.233976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:18.234042 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:18.234124 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:18.235056 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.236033 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.237325 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.237765 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.237880 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:18.240277 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:18.240284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:18.240289 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:18.240329 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:18.242869 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:18.243766 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:18.243772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:18.243775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:18.243940 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:18.244411 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:18.244416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:18.244418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:18.244723 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:18.244872 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:18.246601 (+ 1729us) server_negotiation.cc:299] Negotiation successful
0812 01:11:18.246819 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 01:11:18.254595 23172 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:40373
I20260812 01:11:18.254627 23418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:40373 every 8 connection(s)
I20260812 01:11:18.255793 23172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:11:18.257714 23419 heartbeater.cc:347] Connected to a master server at 127.20.151.62:44023
I20260812 01:11:18.257936 23419 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:18.258575 23419 heartbeater.cc:511] Master 127.20.151.62:44023 requested a full tablet report, sending...
I20260812 01:11:18.260589 23036 ts_manager.cc:194] Registered new tserver with Master: 3f83fcbdc88e4d679690f2d7ac48881c (127.20.151.1:40373)
I20260812 01:11:18.261396 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 23172
I20260812 01:11:18.261601 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:11:18.262125 23036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:40437
I20260812 01:11:18.262221 21084 external_mini_cluster.cc:1428] Setting key 85501c050e70db5677c466f2ef6ff9c4
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:11:18.310496 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:44023
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:18.646472 23426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:18.646777 23426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:18.646924 23426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:18.654379 23426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:18.654522 23426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:18.654645 23426 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 01:11:18.654771 23426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:11:18.664790 23426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.151.62:44023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:18.666360 23426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:18.667852 23426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:18.681138 23434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:18.681111 23435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:18.681319 23437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:18.682025 23426 server_base.cc:1034] running on GCE node
I20260812 01:11:18.682672 23426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:18.683657 23426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:18.684878 23426 hybrid_clock.cc:648] HybridClock initialized: now 1786497078684810 us; error 38 us; skew 500 ppm
I20260812 01:11:18.685173 23426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:18 dist-test-slave-21b9 krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497078, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:18.688508 23426 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:11:18.689991 23426 webserver.cc:460] Webserver started at http://127.20.151.2:38161/ using document root <none> and password file <none>
I20260812 01:11:18.690691 23426 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:18.690784 23426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:18.691098 23426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:18.694113 23426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "28eb02a6b6b444a0922203ab90c2fba7"
format_stamp: "Formatted at 2026-08-12 01:11:18 on dist-test-slave-21b9"
server_key: "bef2aa78b71889697fd70bba38f9f606"
server_key_iv: "fe4db1b056f9ade8dcee3b049ee3cd1a"
server_key_version: "encryptionkey@0"
I20260812 01:11:18.694926 23426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "28eb02a6b6b444a0922203ab90c2fba7"
format_stamp: "Formatted at 2026-08-12 01:11:18 on dist-test-slave-21b9"
server_key: "bef2aa78b71889697fd70bba38f9f606"
server_key_iv: "fe4db1b056f9ade8dcee3b049ee3cd1a"
server_key_version: "encryptionkey@0"
I20260812 01:11:18.700343 23426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 01:11:18.703991 23449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:18.705370 23426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:18.705544 23426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "28eb02a6b6b444a0922203ab90c2fba7"
format_stamp: "Formatted at 2026-08-12 01:11:18 on dist-test-slave-21b9"
server_key: "bef2aa78b71889697fd70bba38f9f606"
server_key_iv: "fe4db1b056f9ade8dcee3b049ee3cd1a"
server_key_version: "encryptionkey@0"
I20260812 01:11:18.705777 23426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:18.727306 23426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:18.732039 23426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:18.732309 23426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:18.735316 23426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:18.735428 23426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:18.735589 23426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:18.735669 23426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 01:11:18 dist-test-slave-21b9 krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497078, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:18.760314 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:18.735974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:58763 (local address 127.20.151.62:44023)
0812 01:11:18.736194 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:18.736201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:18.737810 (+ 1609us) server_negotiation.cc:407] Connection header received
0812 01:11:18.740016 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:18.740022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:18.740114 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:18.740203 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:18.742460 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.743676 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.746225 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.746760 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.746901 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:18.752707 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:18.752713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:18.752717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:18.752764 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:18.755905 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:18.756984 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:18.756990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:18.756993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:18.757186 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:18.757912 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:18.757918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:18.757921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:18.758296 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:18.758472 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:18.759663 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0812 01:11:18.759924 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 01:11:18.761482 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:18.736740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:44023 (local address 127.20.151.2:58763)
0812 01:11:18.737627 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:18.737670 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:18.739708 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:18.740511 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:18.740524 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:18.741375 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:18.742244 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:18.742274 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.743876 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.743883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:18.746068 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:18.746086 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.747046 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.747054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:18.747327 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:18.748330 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:18.748367 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:18.752401 (+ 4034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:18.756109 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:18.756122 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:18.756138 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:18.756824 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:18.757380 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:18.757386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:18.757391 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:18.757759 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:18.758563 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:18.758572 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:18.759193 (+ 621us) client_negotiation.cc:769] Sending connection context
0812 01:11:18.759545 (+ 352us) client_negotiation.cc:240] Negotiation successful
0812 01:11:18.760216 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":396,"threads_started":1}
I20260812 01:11:18.782711 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:18.768328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:51395 (local address 127.20.151.62:44023)
0812 01:11:18.768578 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:18.768585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:18.769234 (+ 649us) server_negotiation.cc:407] Connection header received
0812 01:11:18.769431 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:18.769438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:18.769526 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:18.769628 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:18.770603 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.772032 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.773545 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.774258 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.774388 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:18.776824 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:18.776829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:18.776832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:18.776866 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:18.779863 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:18.780552 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:18.780559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:18.780564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:18.780745 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:18.781210 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:18.781216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:18.781219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:18.781628 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:18.781800 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:18.782304 (+ 504us) server_negotiation.cc:299] Negotiation successful
0812 01:11:18.782473 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 01:11:18.783119 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:18.768400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:44023 (local address 127.20.151.2:51395)
0812 01:11:18.769122 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:18.769140 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:18.769288 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:18.769654 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:18.769658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:18.770148 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:18.770455 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:18.770464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.772180 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.772185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:18.773419 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:18.773434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:18.774453 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:18.774457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:18.774603 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:18.775410 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:18.775427 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:18.776620 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:18.779991 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:18.779996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:18.779999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:18.780434 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:18.780892 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:18.780898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:18.780902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:18.781044 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:18.781775 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:18.781780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:18.782021 (+ 241us) client_negotiation.cc:769] Sending connection context
0812 01:11:18.782201 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 01:11:18.782580 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":373,"threads_started":1}
I20260812 01:11:18.793220 23426 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:40459
I20260812 01:11:18.793251 23674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:40459 every 8 connection(s)
I20260812 01:11:18.794445 23426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:11:18.796483 23676 heartbeater.cc:347] Connected to a master server at 127.20.151.62:44023
I20260812 01:11:18.796715 23676 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:18.797348 23676 heartbeater.cc:511] Master 127.20.151.62:44023 requested a full tablet report, sending...
I20260812 01:11:18.798838 23036 ts_manager.cc:194] Registered new tserver with Master: 28eb02a6b6b444a0922203ab90c2fba7 (127.20.151.2:40459)
I20260812 01:11:18.799738 23036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:58763
I20260812 01:11:18.800716 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 23426
I20260812 01:11:18.800938 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:11:18.801507 21084 external_mini_cluster.cc:1428] Setting key 94d880529d32a34355fd219012d3dc2c
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:11:18.848959 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:44023
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:19.180994 23680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:19.181301 23680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:19.181445 23680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:19.189254 23680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:19.189380 23680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:19.189527 23680 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 01:11:19.189639 23680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:11:19.199667 23680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.151.62:44023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:19.201012 23680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:19.202405 23680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:19.215816 23691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:19.215780 23688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:19.215866 23689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:19.216769 23680 server_base.cc:1034] running on GCE node
I20260812 01:11:19.217435 23680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:19.218482 23680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:19.219720 23680 hybrid_clock.cc:648] HybridClock initialized: now 1786497079219658 us; error 46 us; skew 500 ppm
I20260812 01:11:19.220011 23680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:19 dist-test-slave-21b9 krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497079, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:19.223187 23680 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:11:19.224804 23680 webserver.cc:460] Webserver started at http://127.20.151.3:37823/ using document root <none> and password file <none>
I20260812 01:11:19.225522 23680 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:19.225616 23680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:19.225915 23680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:19.228960 23680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "6ce75f6c019f43bf9112091974418250"
format_stamp: "Formatted at 2026-08-12 01:11:19 on dist-test-slave-21b9"
server_key: "24ee654d7f57fa9ded43556017228fcb"
server_key_iv: "51c523c7a071daf72ad1b2e156f913ea"
server_key_version: "encryptionkey@0"
I20260812 01:11:19.229763 23680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "6ce75f6c019f43bf9112091974418250"
format_stamp: "Formatted at 2026-08-12 01:11:19 on dist-test-slave-21b9"
server_key: "24ee654d7f57fa9ded43556017228fcb"
server_key_iv: "51c523c7a071daf72ad1b2e156f913ea"
server_key_version: "encryptionkey@0"
I20260812 01:11:19.235008 23680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:11:19.238611 23704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:19.240017 23680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:11:19.240185 23680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "6ce75f6c019f43bf9112091974418250"
format_stamp: "Formatted at 2026-08-12 01:11:19 on dist-test-slave-21b9"
server_key: "24ee654d7f57fa9ded43556017228fcb"
server_key_iv: "51c523c7a071daf72ad1b2e156f913ea"
server_key_version: "encryptionkey@0"
I20260812 01:11:19.240407 23680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:19.262795 23680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:19.265259 23419 heartbeater.cc:503] Master 127.20.151.62:44023 was elected leader, sending a full tablet report...
I20260812 01:11:19.267657 23680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:19.267894 23680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:19.270599 23680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:19.270685 23680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:19.270819 23680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:19.270881 23680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 01:11:19 dist-test-slave-21b9 krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497079, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:19.297572 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.271308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:55787 (local address 127.20.151.62:44023)
0812 01:11:19.271498 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:19.271505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:19.273155 (+ 1650us) server_negotiation.cc:407] Connection header received
0812 01:11:19.275558 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:19.275564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:19.275654 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:19.275748 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:19.277455 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.278221 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.280144 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.280712 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.280842 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.289227 (+ 8385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:19.289235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:19.289241 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:19.289287 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:19.292751 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:19.294079 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:19.294085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:19.294088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:19.294282 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:19.294992 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:19.294997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:19.295001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:19.295447 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:19.295624 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:19.296843 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0812 01:11:19.297158 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 01:11:19.298300 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.272004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:44023 (local address 127.20.151.3:55787)
0812 01:11:19.272977 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:19.273023 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:19.275153 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:19.276003 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:19.276011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:19.276629 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:19.277266 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.277286 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.278378 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.278382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.280002 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.280020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.281013 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.281022 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.281376 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.285015 (+ 3639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:19.285062 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:19.288872 (+ 3810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:19.292959 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:19.292975 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:19.292998 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:19.293855 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:19.294443 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:19.294449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:19.294453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:19.294834 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:19.295602 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:19.295614 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:19.296343 (+ 729us) client_negotiation.cc:769] Sending connection context
0812 01:11:19.296619 (+ 276us) client_negotiation.cc:240] Negotiation successful
0812 01:11:19.297070 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":450,"threads_started":1}
I20260812 01:11:19.319368 23680 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:40645
I20260812 01:11:19.319396 23920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:40645 every 8 connection(s)
I20260812 01:11:19.321049 23680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:11:19.321938 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.306448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:41811 (local address 127.20.151.62:44023)
0812 01:11:19.306664 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:19.306670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:19.307949 (+ 1279us) server_negotiation.cc:407] Connection header received
0812 01:11:19.308145 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:19.308149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:19.308226 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:19.308317 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:19.309403 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.310328 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.311755 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.312381 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.312526 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.315230 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:19.315237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:19.315242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:19.315303 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:19.318627 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:19.319670 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:19.319676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:19.319679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:19.319848 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:19.320304 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:19.320309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:19.320312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:19.320714 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:19.320868 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:19.321320 (+ 452us) server_negotiation.cc:299] Negotiation successful
0812 01:11:19.321586 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 01:11:19.321950 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.306471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:44023 (local address 127.20.151.3:41811)
0812 01:11:19.307823 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:19.307846 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:19.308016 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:19.308383 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:19.308387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:19.308932 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:19.309244 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.309255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.310487 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.310492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.311617 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.311630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.312619 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.312626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.312854 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.313666 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:19.313697 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:19.315008 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:19.318795 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:19.318802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:19.318809 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:19.319506 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:19.320001 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:19.320007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:19.320010 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:19.320169 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:19.320849 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:19.320853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:19.321033 (+ 180us) client_negotiation.cc:769] Sending connection context
0812 01:11:19.321159 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 01:11:19.321471 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"thread_start_us":1007,"threads_started":1}
I20260812 01:11:19.323287 23925 heartbeater.cc:347] Connected to a master server at 127.20.151.62:44023
I20260812 01:11:19.323520 23925 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:19.324414 23925 heartbeater.cc:511] Master 127.20.151.62:44023 requested a full tablet report, sending...
I20260812 01:11:19.326032 23036 ts_manager.cc:194] Registered new tserver with Master: 6ce75f6c019f43bf9112091974418250 (127.20.151.3:40645)
I20260812 01:11:19.326958 23036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:55787
I20260812 01:11:19.330540 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 23680
I20260812 01:11:19.330726 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:11:19.331157 21084 external_mini_cluster.cc:1428] Setting key 0ec44f67557dd0b7c7697f4a3d08a5e1
I20260812 01:11:19.334093 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 01:11:19 dist-test-slave-21b9 krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 01:11:19 dist-test-slave-21b9 krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:19.388381 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.374557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.20.151.62:44023)
0812 01:11:19.374753 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:19.374757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:19.375047 (+ 290us) server_negotiation.cc:407] Connection header received
0812 01:11:19.375210 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:19.375215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:19.375321 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:19.375446 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:19.376840 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.377669 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.378914 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.379419 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.379541 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.383365 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:19.383370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:19.383374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:19.383418 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:19.385926 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:19.386536 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:19.386541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:19.386543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:19.386681 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:19.387032 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:19.387037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:19.387039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:19.387363 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:19.387556 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:19.387825 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 01:11:19.388012 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 01:11:19.392623 23036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:11:19.399981 23036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:11:19.424855 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.413823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.20.151.3:40645)
0812 01:11:19.414988 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:19.414998 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:19.416206 (+ 1208us) server_negotiation.cc:407] Connection header received
0812 01:11:19.416555 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:19.416566 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:19.416855 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:19.417034 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:19.418212 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.419838 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.422104 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.423425 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.423557 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.423799 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:19.424313 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 01:11:19.424498 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":777,"threads_started":1}
I20260812 01:11:19.425096 23953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.413109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.20.151.2:40459)
0812 01:11:19.417251 (+ 4142us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:19.417259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:19.417287 (+ 28us) server_negotiation.cc:407] Connection header received
0812 01:11:19.417385 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:19.417392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:19.417616 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:19.417849 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:19.418506 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.419946 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.422683 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.424053 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.424206 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.424407 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:19.424624 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 01:11:19.424793 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3970,"thread_start_us":3837,"threads_started":1}
I20260812 01:11:19.425179 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.412867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:40459 (local address 127.0.0.1:48388)
0812 01:11:19.413831 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:19.413886 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:19.414065 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:19.417837 (+ 3772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:19.417842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:19.417869 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:19.418284 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.418296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.420061 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.420066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.422486 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.422502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.424268 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.424274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.424438 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.424477 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 01:11:19.424569 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 01:11:19.424772 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":80,"thread_start_us":518,"threads_started":1}
I20260812 01:11:19.426059 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.414211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:40373 (local address 127.0.0.1:32788)
0812 01:11:19.417594 (+ 3383us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:19.417615 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:19.417757 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:19.418354 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:19.418357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:19.418382 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:19.418732 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.418743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.420858 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.420867 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.423038 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.423053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.425136 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.425143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.425331 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.425360 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 01:11:19.425443 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 01:11:19.425575 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3249,"mutex_wait_us":3,"thread_start_us":1481,"threads_started":1}
I20260812 01:11:19.426143 23956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.414722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.20.151.1:40373)
0812 01:11:19.415500 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:19.415509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:19.417744 (+ 2235us) server_negotiation.cc:407] Connection header received
0812 01:11:19.417970 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:19.417977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:19.418158 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:19.418353 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:19.419086 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.420706 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.423222 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.424658 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.424784 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.424969 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:19.425707 (+ 738us) server_negotiation.cc:299] Negotiation successful
0812 01:11:19.425872 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":439,"threads_started":1}
I20260812 01:11:19.426103 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.413052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:40645 (local address 127.0.0.1:51676)
0812 01:11:19.416016 (+ 2964us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:19.416045 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:19.416354 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:19.417475 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:19.417483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:19.417514 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:19.417913 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.417931 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.420023 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.420028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.421937 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.421950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.423913 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.423917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.424061 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.424094 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 01:11:19.425422 (+ 1328us) client_negotiation.cc:240] Negotiation successful
0812 01:11:19.425583 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2785,"mutex_wait_us":98,"thread_start_us":4255,"threads_started":1}
I20260812 01:11:19.429605 23796 tablet_service.cc:1467] Processing CreateTablet for tablet fa8ec83b580248f49effed3e7dcd48a1 (DEFAULT_TABLE table=test-table [id=9fef6dfcb2924342a8102d07ae1050ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:19.429538 23294 tablet_service.cc:1467] Processing CreateTablet for tablet fa8ec83b580248f49effed3e7dcd48a1 (DEFAULT_TABLE table=test-table [id=9fef6dfcb2924342a8102d07ae1050ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:19.429534 23553 tablet_service.cc:1467] Processing CreateTablet for tablet fa8ec83b580248f49effed3e7dcd48a1 (DEFAULT_TABLE table=test-table [id=9fef6dfcb2924342a8102d07ae1050ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:19.430852 23796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8ec83b580248f49effed3e7dcd48a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:19.430909 23294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8ec83b580248f49effed3e7dcd48a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:19.430912 23553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8ec83b580248f49effed3e7dcd48a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:19.443056 23958 tablet_bootstrap.cc:492] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: Bootstrap starting.
I20260812 01:11:19.446095 23960 tablet_bootstrap.cc:492] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: Bootstrap starting.
I20260812 01:11:19.447817 23959 tablet_bootstrap.cc:492] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: Bootstrap starting.
I20260812 01:11:19.449378 23958 tablet_bootstrap.cc:654] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:19.450798 23960 tablet_bootstrap.cc:654] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:19.451247 23958 log.cc:824] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:19.451619 23959 tablet_bootstrap.cc:654] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:19.452402 23960 log.cc:824] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:19.453079 23959 log.cc:824] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:19.455173 23958 tablet_bootstrap.cc:492] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: No bootstrap required, opened a new log
I20260812 01:11:19.455586 23958 ts_tablet_manager.cc:1397] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260812 01:11:19.456048 23960 tablet_bootstrap.cc:492] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: No bootstrap required, opened a new log
I20260812 01:11:19.456396 23960 ts_tablet_manager.cc:1397] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260812 01:11:19.456492 23959 tablet_bootstrap.cc:492] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: No bootstrap required, opened a new log
I20260812 01:11:19.456689 23959 ts_tablet_manager.cc:1397] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 01:11:19.466063 23959 raft_consensus.cc:348] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } }
I20260812 01:11:19.466950 23959 raft_consensus.cc:374] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:19.467110 23959 raft_consensus.cc:729] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f83fcbdc88e4d679690f2d7ac48881c, State: Initialized, Role: FOLLOWER
I20260812 01:11:19.467864 23959 consensus_queue.cc:260] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } }
I20260812 01:11:19.468856 23958 raft_consensus.cc:348] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } }
I20260812 01:11:19.468856 23960 raft_consensus.cc:348] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } }
I20260812 01:11:19.469499 23958 raft_consensus.cc:374] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:19.469551 23960 raft_consensus.cc:374] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:19.469679 23958 raft_consensus.cc:729] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce75f6c019f43bf9112091974418250, State: Initialized, Role: FOLLOWER
I20260812 01:11:19.469758 23960 raft_consensus.cc:729] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb02a6b6b444a0922203ab90c2fba7, State: Initialized, Role: FOLLOWER
I20260812 01:11:19.469954 23959 ts_tablet_manager.cc:1428] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260812 01:11:19.470566 23960 consensus_queue.cc:260] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } }
I20260812 01:11:19.470551 23958 consensus_queue.cc:260] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } }
I20260812 01:11:19.472138 23676 heartbeater.cc:503] Master 127.20.151.62:44023 was elected leader, sending a full tablet report...
I20260812 01:11:19.472698 23960 ts_tablet_manager.cc:1428] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260812 01:11:19.473881 23925 heartbeater.cc:503] Master 127.20.151.62:44023 was elected leader, sending a full tablet report...
I20260812 01:11:19.474527 23958 ts_tablet_manager.cc:1428] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: Time spent starting tablet: real 0.019s user 0.007s sys 0.011s
W20260812 01:11:19.508178 23420 tablet.cc:2369] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:19.522899 23973 raft_consensus.cc:482] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:11:19.523311 23973 raft_consensus.cc:504] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } }
I20260812 01:11:19.525414 23973 leader_election.cc:288] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28eb02a6b6b444a0922203ab90c2fba7 (127.20.151.2:40459), 3f83fcbdc88e4d679690f2d7ac48881c (127.20.151.1:40373)
I20260812 01:11:19.533013 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.525782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:40459 (local address 127.20.151.3:59321)
0812 01:11:19.525977 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:19.525997 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:19.526134 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:19.526708 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:19.526713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:19.526749 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:19.527056 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.527081 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.528550 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.528556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.530367 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.530380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.532145 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.532151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.532402 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.532429 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 01:11:19.532524 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 01:11:19.532664 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":3}
I20260812 01:11:19.533054 23953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.525720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:59321 (local address 127.20.151.2:40459)
0812 01:11:19.525948 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:19.525953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:19.526062 (+ 109us) server_negotiation.cc:407] Connection header received
0812 01:11:19.526307 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:19.526314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:19.526401 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:19.526505 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:19.527251 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.528410 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.530547 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.531996 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.532148 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.532226 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:19.532575 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 01:11:19.532708 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 01:11:19.534055 23956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.526501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:56827 (local address 127.20.151.1:40373)
0812 01:11:19.526738 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:19.526744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:19.527191 (+ 447us) server_negotiation.cc:407] Connection header received
0812 01:11:19.527387 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:19.527390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:19.527464 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:19.527535 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:19.528186 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.529320 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.531952 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.533057 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.533264 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.533334 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:19.533752 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 01:11:19.533876 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 01:11:19.534425 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.526149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:40373 (local address 127.20.151.3:56827)
0812 01:11:19.527082 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:19.527097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:19.527254 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:19.527579 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:19.527584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:19.527610 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:19.528044 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.528058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.529488 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.529493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.531755 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.531773 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.533399 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.533406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.533588 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.533613 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:11:19.533690 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 01:11:19.533790 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":451,"threads_started":1}
I20260812 01:11:19.534529 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8ec83b580248f49effed3e7dcd48a1" candidate_uuid: "6ce75f6c019f43bf9112091974418250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb02a6b6b444a0922203ab90c2fba7" is_pre_election: true
I20260812 01:11:19.535161 23584 raft_consensus.cc:2454] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce75f6c019f43bf9112091974418250 in term 0.
I20260812 01:11:19.535635 23330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8ec83b580248f49effed3e7dcd48a1" candidate_uuid: "6ce75f6c019f43bf9112091974418250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" is_pre_election: true
I20260812 01:11:19.536036 23717 leader_election.cc:302] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb02a6b6b444a0922203ab90c2fba7, 6ce75f6c019f43bf9112091974418250; no voters:
I20260812 01:11:19.536440 23973 raft_consensus.cc:2781] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:11:19.536413 23330 raft_consensus.cc:2454] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce75f6c019f43bf9112091974418250 in term 0.
I20260812 01:11:19.536616 23973 raft_consensus.cc:482] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:11:19.536798 23973 raft_consensus.cc:3037] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:19.539655 23973 raft_consensus.cc:504] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } }
I20260812 01:11:19.540549 23973 leader_election.cc:288] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [CANDIDATE]: Term 1 election: Requested vote from peers 28eb02a6b6b444a0922203ab90c2fba7 (127.20.151.2:40459), 3f83fcbdc88e4d679690f2d7ac48881c (127.20.151.1:40373)
I20260812 01:11:19.541046 23584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8ec83b580248f49effed3e7dcd48a1" candidate_uuid: "6ce75f6c019f43bf9112091974418250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eb02a6b6b444a0922203ab90c2fba7"
I20260812 01:11:19.541144 23330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8ec83b580248f49effed3e7dcd48a1" candidate_uuid: "6ce75f6c019f43bf9112091974418250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f83fcbdc88e4d679690f2d7ac48881c"
I20260812 01:11:19.541352 23330 raft_consensus.cc:3037] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:19.541355 23584 raft_consensus.cc:3037] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:19.545318 23330 raft_consensus.cc:2454] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce75f6c019f43bf9112091974418250 in term 1.
I20260812 01:11:19.545320 23584 raft_consensus.cc:2454] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce75f6c019f43bf9112091974418250 in term 1.
I20260812 01:11:19.545728 23717 leader_election.cc:302] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb02a6b6b444a0922203ab90c2fba7, 6ce75f6c019f43bf9112091974418250; no voters:
I20260812 01:11:19.546008 23973 raft_consensus.cc:2781] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 1 FOLLOWER]: Leader election won for term 1
W20260812 01:11:19.546614 23677 tablet.cc:2369] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:19.546731 23973 raft_consensus.cc:686] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 1 LEADER]: Becoming Leader. State: Replica: 6ce75f6c019f43bf9112091974418250, State: Running, Role: LEADER
I20260812 01:11:19.547380 23973 consensus_queue.cc:237] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } }
I20260812 01:11:19.552253 23034 catalog_manager.cc:5495] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ce75f6c019f43bf9112091974418250 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "6ce75f6c019f43bf9112091974418250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } health_report { overall_health: UNKNOWN } } }
W20260812 01:11:19.573514 23927 tablet.cc:2369] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:19.590595 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.584402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.20.151.3:40645)
0812 01:11:19.584588 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:19.584593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:19.584853 (+ 260us) server_negotiation.cc:407] Connection header received
0812 01:11:19.585105 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:19.585109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:19.585189 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:19.585309 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:19.585750 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.586607 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.588202 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.588856 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.588953 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.589515 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:19.590043 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:19.590152 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:19.590235 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 01:11:19.590304 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 01:11:19.600800 23584 raft_consensus.cc:1261] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 FOLLOWER]: Refusing update from remote peer 6ce75f6c019f43bf9112091974418250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:19.600842 23330 raft_consensus.cc:1261] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c [term 1 FOLLOWER]: Refusing update from remote peer 6ce75f6c019f43bf9112091974418250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:19.601845 23977 consensus_queue.cc:1035] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:19.602403 23973 consensus_queue.cc:1035] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:19.615157 23981 mvcc.cc:205] Tried to move back new op lower bound from 7317492038033121280 to 7317492037830115328. Current Snapshot: MvccSnapshot[applied={T|T < 7317492038033121280}]
I20260812 01:11:19.620273 23984 mvcc.cc:205] Tried to move back new op lower bound from 7317492038033121280 to 7317492037830115328. Current Snapshot: MvccSnapshot[applied={T|T < 7317492038033121280}]
I20260812 01:11:19.647588 23036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:19.647879 23036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:19.652274 23036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138: Sending DeleteTablet for 3 replicas of tablet fa8ec83b580248f49effed3e7dcd48a1
I20260812 01:11:19.653683 23553 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8ec83b580248f49effed3e7dcd48a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:19 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:48388
I20260812 01:11:19.653690 23796 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8ec83b580248f49effed3e7dcd48a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:19 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:51676
I20260812 01:11:19.654598 23294 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8ec83b580248f49effed3e7dcd48a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:19 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:32788
I20260812 01:11:19.656414 23036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 01:11:19.657140 24009 tablet_replica.cc:330] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: stopping tablet replica
W20260812 01:11:19.657433 23036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:11:19.657739 24009 raft_consensus.cc:2229] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:19.658448 24009 raft_consensus.cc:2258] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:19.659770 24008 tablet_replica.cc:330] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: stopping tablet replica
I20260812 01:11:19.660368 24008 raft_consensus.cc:2229] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:19.661139 24008 raft_consensus.cc:2258] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:19.661755 24009 ts_tablet_manager.cc:1905] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:19.663570 24005 tablet_replica.cc:330] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: stopping tablet replica
I20260812 01:11:19.664089 24005 raft_consensus.cc:2229] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:11:19.665016 24005 raft_consensus.cc:2258] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:19.665616 24008 ts_tablet_manager.cc:1905] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:19.667668 23796 tablet_service.cc:1467] Processing CreateTablet for tablet 088b68e106ec47ee8925da7ca44eab03 (DEFAULT_TABLE table=test-table [id=3f572b0eaf534dd592b86972e4f23049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:19.667933 23553 tablet_service.cc:1467] Processing CreateTablet for tablet 088b68e106ec47ee8925da7ca44eab03 (DEFAULT_TABLE table=test-table [id=3f572b0eaf534dd592b86972e4f23049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:19.668200 23796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088b68e106ec47ee8925da7ca44eab03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:19.668682 23294 tablet_service.cc:1467] Processing CreateTablet for tablet 088b68e106ec47ee8925da7ca44eab03 (DEFAULT_TABLE table=test-table [id=3f572b0eaf534dd592b86972e4f23049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:19.669271 23294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088b68e106ec47ee8925da7ca44eab03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:19.671525 24009 ts_tablet_manager.cc:1918] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:11:19.671838 24009 log.cc:1196] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/fa8ec83b580248f49effed3e7dcd48a1
I20260812 01:11:19.673527 23553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088b68e106ec47ee8925da7ca44eab03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:19.676369 23958 tablet_bootstrap.cc:492] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250: Bootstrap starting.
I20260812 01:11:19.678758 23958 tablet_bootstrap.cc:654] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:19.680052 23959 tablet_bootstrap.cc:492] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c: Bootstrap starting.
I20260812 01:11:19.680802 23960 tablet_bootstrap.cc:492] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7: Bootstrap starting.
I20260812 01:11:19.681833 24009 ts_tablet_manager.cc:1939] T fa8ec83b580248f49effed3e7dcd48a1 P 3f83fcbdc88e4d679690f2d7ac48881c: Deleting consensus metadata
I20260812 01:11:19.682277 23959 tablet_bootstrap.cc:654] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:19.682958 23960 tablet_bootstrap.cc:654] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:19.686228 23014 catalog_manager.cc:4841] TS 3f83fcbdc88e4d679690f2d7ac48881c (127.20.151.1:40373): tablet fa8ec83b580248f49effed3e7dcd48a1 (table test-table [id=9fef6dfcb2924342a8102d07ae1050ce]) successfully deleted
I20260812 01:11:19.688876 23960 tablet_bootstrap.cc:492] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7: No bootstrap required, opened a new log
I20260812 01:11:19.689118 23960 ts_tablet_manager.cc:1397] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260812 01:11:19.690066 23960 raft_consensus.cc:348] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:19.690315 23960 raft_consensus.cc:374] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:19.690425 23960 raft_consensus.cc:729] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb02a6b6b444a0922203ab90c2fba7, State: Initialized, Role: FOLLOWER
I20260812 01:11:19.690770 23960 consensus_queue.cc:260] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:19.692031 23960 ts_tablet_manager.cc:1428] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260812 01:11:19.692907 24005 ts_tablet_manager.cc:1905] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:19.694394 23958 tablet_bootstrap.cc:492] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250: No bootstrap required, opened a new log
I20260812 01:11:19.694669 23958 ts_tablet_manager.cc:1397] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20260812 01:11:19.695483 24008 ts_tablet_manager.cc:1918] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:11:19.696043 24008 log.cc:1196] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/fa8ec83b580248f49effed3e7dcd48a1
I20260812 01:11:19.696290 23958 raft_consensus.cc:348] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:19.696583 23958 raft_consensus.cc:374] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:19.696794 23958 raft_consensus.cc:729] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce75f6c019f43bf9112091974418250, State: Initialized, Role: FOLLOWER
I20260812 01:11:19.697173 23958 consensus_queue.cc:260] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:19.697979 23958 ts_tablet_manager.cc:1428] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 01:11:19.698290 24008 ts_tablet_manager.cc:1939] T fa8ec83b580248f49effed3e7dcd48a1 P 28eb02a6b6b444a0922203ab90c2fba7: Deleting consensus metadata
I20260812 01:11:19.698911 23959 tablet_bootstrap.cc:492] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c: No bootstrap required, opened a new log
I20260812 01:11:19.699153 23959 ts_tablet_manager.cc:1397] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20260812 01:11:19.700548 23959 raft_consensus.cc:348] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:19.701036 23959 raft_consensus.cc:374] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:19.701268 23959 raft_consensus.cc:729] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f83fcbdc88e4d679690f2d7ac48881c, State: Initialized, Role: FOLLOWER
I20260812 01:11:19.701287 23014 catalog_manager.cc:4841] TS 28eb02a6b6b444a0922203ab90c2fba7 (127.20.151.2:40459): tablet fa8ec83b580248f49effed3e7dcd48a1 (table test-table [id=9fef6dfcb2924342a8102d07ae1050ce]) successfully deleted
I20260812 01:11:19.701757 23959 consensus_queue.cc:260] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:19.702658 23959 ts_tablet_manager.cc:1428] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 01:11:19.706478 24005 ts_tablet_manager.cc:1918] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:11:19.706682 24005 log.cc:1196] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/fa8ec83b580248f49effed3e7dcd48a1
I20260812 01:11:19.707477 24005 ts_tablet_manager.cc:1939] T fa8ec83b580248f49effed3e7dcd48a1 P 6ce75f6c019f43bf9112091974418250: Deleting consensus metadata
I20260812 01:11:19.709499 23011 catalog_manager.cc:4841] TS 6ce75f6c019f43bf9112091974418250 (127.20.151.3:40645): tablet fa8ec83b580248f49effed3e7dcd48a1 (table test-table [id=9fef6dfcb2924342a8102d07ae1050ce]) successfully deleted
I20260812 01:11:19.960238 23971 raft_consensus.cc:482] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:11:19.960475 23971 raft_consensus.cc:504] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:19.962126 23971 leader_election.cc:288] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f83fcbdc88e4d679690f2d7ac48881c (127.20.151.1:40373), 6ce75f6c019f43bf9112091974418250 (127.20.151.3:40645)
I20260812 01:11:19.973214 23956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.962767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:50823 (local address 127.20.151.1:40373)
0812 01:11:19.962967 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:19.962974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:19.963513 (+ 539us) server_negotiation.cc:407] Connection header received
0812 01:11:19.963841 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:19.963847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:19.963950 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:19.964037 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:19.965807 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.966733 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.970142 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.971657 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.971883 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.971967 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:19.972652 (+ 685us) server_negotiation.cc:299] Negotiation successful
0812 01:11:19.972799 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 01:11:19.973419 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.962563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:40373 (local address 127.20.151.2:50823)
0812 01:11:19.963392 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:19.963413 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:19.963643 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:19.964303 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:19.964308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:19.964362 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:19.965521 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.965538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.966874 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.966879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.969879 (+ 3000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.969898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.972076 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.972089 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.972373 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.972502 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 01:11:19.972575 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 01:11:19.972730 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":8,"spinlock_wait_cycles":5888,"thread_start_us":414,"threads_started":1}
I20260812 01:11:19.973424 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.962717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:40645 (local address 127.20.151.2:36453)
0812 01:11:19.964109 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:19.964128 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:19.964360 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:19.964730 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:19.964740 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:19.964768 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:19.965700 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.965717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.967498 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.967504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.970154 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:19.970169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.972183 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.972192 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:19.972443 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.972478 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 01:11:19.972580 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 01:11:19.972730 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"mutex_wait_us":5,"spinlock_wait_cycles":2944,"thread_start_us":1274,"threads_started":1}
I20260812 01:11:19.974282 23330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088b68e106ec47ee8925da7ca44eab03" candidate_uuid: "28eb02a6b6b444a0922203ab90c2fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" is_pre_election: true
I20260812 01:11:19.974550 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:19.963756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:36453 (local address 127.20.151.3:40645)
0812 01:11:19.964003 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:19.964009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:19.964220 (+ 211us) server_negotiation.cc:407] Connection header received
0812 01:11:19.964532 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:19.964536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:19.964599 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:19.964696 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:19.965928 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.967347 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.970330 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:19.971763 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:19.971920 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:19.971995 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:19.974079 (+ 2084us) server_negotiation.cc:299] Negotiation successful
0812 01:11:19.974226 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 01:11:19.974534 23330 raft_consensus.cc:2454] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28eb02a6b6b444a0922203ab90c2fba7 in term 0.
I20260812 01:11:19.975471 23457 leader_election.cc:302] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb02a6b6b444a0922203ab90c2fba7, 3f83fcbdc88e4d679690f2d7ac48881c; no voters:
I20260812 01:11:19.975689 23830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088b68e106ec47ee8925da7ca44eab03" candidate_uuid: "28eb02a6b6b444a0922203ab90c2fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce75f6c019f43bf9112091974418250" is_pre_election: true
I20260812 01:11:19.976140 23971 raft_consensus.cc:2781] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:11:19.976286 23971 raft_consensus.cc:482] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:11:19.976213 23830 raft_consensus.cc:2454] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28eb02a6b6b444a0922203ab90c2fba7 in term 0.
I20260812 01:11:19.976491 23971 raft_consensus.cc:3037] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:19.980194 23971 raft_consensus.cc:504] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:19.981076 23971 leader_election.cc:288] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [CANDIDATE]: Term 1 election: Requested vote from peers 3f83fcbdc88e4d679690f2d7ac48881c (127.20.151.1:40373), 6ce75f6c019f43bf9112091974418250 (127.20.151.3:40645)
I20260812 01:11:19.981602 23830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088b68e106ec47ee8925da7ca44eab03" candidate_uuid: "28eb02a6b6b444a0922203ab90c2fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce75f6c019f43bf9112091974418250"
I20260812 01:11:19.981653 23330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088b68e106ec47ee8925da7ca44eab03" candidate_uuid: "28eb02a6b6b444a0922203ab90c2fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f83fcbdc88e4d679690f2d7ac48881c"
I20260812 01:11:19.981855 23830 raft_consensus.cc:3037] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:19.981914 23330 raft_consensus.cc:3037] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:19.985520 23330 raft_consensus.cc:2454] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb02a6b6b444a0922203ab90c2fba7 in term 1.
I20260812 01:11:19.985606 23830 raft_consensus.cc:2454] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb02a6b6b444a0922203ab90c2fba7 in term 1.
I20260812 01:11:19.986080 23457 leader_election.cc:302] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb02a6b6b444a0922203ab90c2fba7, 3f83fcbdc88e4d679690f2d7ac48881c; no voters:
I20260812 01:11:19.986426 23971 raft_consensus.cc:2781] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:19.987211 23971 raft_consensus.cc:686] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 LEADER]: Becoming Leader. State: Replica: 28eb02a6b6b444a0922203ab90c2fba7, State: Running, Role: LEADER
I20260812 01:11:19.987835 23971 consensus_queue.cc:237] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:19.991935 23034 catalog_manager.cc:5495] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28eb02a6b6b444a0922203ab90c2fba7 (127.20.151.2). New cstate: current_term: 1 leader_uuid: "28eb02a6b6b444a0922203ab90c2fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } health_report { overall_health: UNKNOWN } } }
Aug 12 01:11:20 dist-test-slave-21b9 krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497077, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:20.118831 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.098039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:44023 (local address 127.0.0.1:42748)
0812 01:11:20.099784 (+ 1745us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:20.099801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:20.099961 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:20.100272 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:20.100277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:20.101232 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:20.101561 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:20.101574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.103306 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.103312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:20.104700 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:20.104715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.105799 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.105804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:20.105934 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.106714 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:20.106740 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:20.110168 (+ 3428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:20.115727 (+ 5559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:20.115746 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:20.115755 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:20.116266 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:20.116806 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:20.116809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:20.116811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:20.117026 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:20.117783 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:20.117789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:20.117950 (+ 161us) client_negotiation.cc:769] Sending connection context
0812 01:11:20.118199 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 01:11:20.118429 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1536,"thread_start_us":1352,"threads_started":1}
I20260812 01:11:20.119251 24029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.098027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.20.151.62:44023)
0812 01:11:20.098887 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:20.098894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:20.099892 (+ 998us) server_negotiation.cc:407] Connection header received
0812 01:11:20.100073 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:20.100076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:20.100139 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:20.100230 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:20.101794 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.103074 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.104865 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.105612 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.105768 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.110418 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:20.110425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:20.110431 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:20.110486 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:20.115540 (+ 5054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.116454 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.116461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.116465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.116678 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.117175 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.117182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.117186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.117644 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:20.117820 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:20.118376 (+ 556us) server_negotiation.cc:299] Negotiation successful
0812 01:11:20.118651 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":378,"threads_started":1}
I20260812 01:11:20.125528 23034 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:42748:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 01:11:20.126608 23034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:11:20.133787 23294 tablet_service.cc:1467] Processing CreateTablet for tablet e0945084ec4847138ef3dbea029b667d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=60f6558586984e8caa9904fbfd5a869d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 01:11:20.133975 23553 tablet_service.cc:1467] Processing CreateTablet for tablet e0945084ec4847138ef3dbea029b667d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=60f6558586984e8caa9904fbfd5a869d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 01:11:20.134377 23294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0945084ec4847138ef3dbea029b667d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:20.134577 23553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0945084ec4847138ef3dbea029b667d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:20.137811 23796 tablet_service.cc:1467] Processing CreateTablet for tablet e0945084ec4847138ef3dbea029b667d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=60f6558586984e8caa9904fbfd5a869d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 01:11:20.138401 23796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0945084ec4847138ef3dbea029b667d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:20.141764 23959 tablet_bootstrap.cc:492] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c: Bootstrap starting.
I20260812 01:11:20.143522 23960 tablet_bootstrap.cc:492] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: Bootstrap starting.
I20260812 01:11:20.144470 23959 tablet_bootstrap.cc:654] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:20.145637 23960 tablet_bootstrap.cc:654] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:20.147182 23958 tablet_bootstrap.cc:492] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250: Bootstrap starting.
I20260812 01:11:20.149657 23958 tablet_bootstrap.cc:654] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:20.150872 23960 tablet_bootstrap.cc:492] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: No bootstrap required, opened a new log
I20260812 01:11:20.151081 23960 ts_tablet_manager.cc:1397] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20260812 01:11:20.152129 23960 raft_consensus.cc:348] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:20.152340 23958 tablet_bootstrap.cc:492] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250: No bootstrap required, opened a new log
I20260812 01:11:20.152411 23960 raft_consensus.cc:374] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:20.152508 23958 ts_tablet_manager.cc:1397] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260812 01:11:20.152514 23960 raft_consensus.cc:729] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eb02a6b6b444a0922203ab90c2fba7, State: Initialized, Role: FOLLOWER
I20260812 01:11:20.152792 23960 consensus_queue.cc:260] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:20.153592 23958 raft_consensus.cc:348] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:20.153898 23959 tablet_bootstrap.cc:492] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c: No bootstrap required, opened a new log
I20260812 01:11:20.153885 23958 raft_consensus.cc:374] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:20.154028 23958 raft_consensus.cc:729] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce75f6c019f43bf9112091974418250, State: Initialized, Role: FOLLOWER
I20260812 01:11:20.154109 23959 ts_tablet_manager.cc:1397] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260812 01:11:20.154402 23958 consensus_queue.cc:260] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:20.154821 23977 tablet_replica.cc:439] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } }
I20260812 01:11:20.155150 23977 tablet_replica.cc:442] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:20.155011 23959 raft_consensus.cc:348] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:20.155333 23959 raft_consensus.cc:374] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:20.155532 23959 raft_consensus.cc:729] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f83fcbdc88e4d679690f2d7ac48881c, State: Initialized, Role: FOLLOWER
I20260812 01:11:20.155535 23971 tablet_replica.cc:439] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } }
I20260812 01:11:20.156028 23971 tablet_replica.cc:442] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:20.155946 23959 consensus_queue.cc:260] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:20.156530 23968 tablet_replica.cc:439] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } }
I20260812 01:11:20.157081 23968 tablet_replica.cc:442] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:20.157517 23960 ts_tablet_manager.cc:1428] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260812 01:11:20.159232 23959 ts_tablet_manager.cc:1428] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260812 01:11:20.160111 23958 ts_tablet_manager.cc:1428] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260812 01:11:20.378810 24020 raft_consensus.cc:482] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:11:20.379029 24020 raft_consensus.cc:504] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:20.379778 24020 leader_election.cc:288] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f83fcbdc88e4d679690f2d7ac48881c (127.20.151.1:40373), 6ce75f6c019f43bf9112091974418250 (127.20.151.3:40645)
I20260812 01:11:20.380327 23330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0945084ec4847138ef3dbea029b667d" candidate_uuid: "28eb02a6b6b444a0922203ab90c2fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" is_pre_election: true
I20260812 01:11:20.380385 23828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0945084ec4847138ef3dbea029b667d" candidate_uuid: "28eb02a6b6b444a0922203ab90c2fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce75f6c019f43bf9112091974418250" is_pre_election: true
I20260812 01:11:20.380605 23330 raft_consensus.cc:2454] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28eb02a6b6b444a0922203ab90c2fba7 in term 0.
I20260812 01:11:20.380604 23828 raft_consensus.cc:2454] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28eb02a6b6b444a0922203ab90c2fba7 in term 0.
I20260812 01:11:20.380865 24020 consensus_queue.cc:1035] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:20.381062 23450 leader_election.cc:302] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb02a6b6b444a0922203ab90c2fba7, 6ce75f6c019f43bf9112091974418250; no voters:
I20260812 01:11:20.381381 23971 raft_consensus.cc:2781] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:11:20.381511 23971 raft_consensus.cc:482] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:11:20.381609 23971 raft_consensus.cc:3037] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:20.384627 23971 raft_consensus.cc:504] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:20.385337 23971 leader_election.cc:288] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [CANDIDATE]: Term 1 election: Requested vote from peers 3f83fcbdc88e4d679690f2d7ac48881c (127.20.151.1:40373), 6ce75f6c019f43bf9112091974418250 (127.20.151.3:40645)
I20260812 01:11:20.385807 23828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0945084ec4847138ef3dbea029b667d" candidate_uuid: "28eb02a6b6b444a0922203ab90c2fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce75f6c019f43bf9112091974418250"
I20260812 01:11:20.385936 23330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0945084ec4847138ef3dbea029b667d" candidate_uuid: "28eb02a6b6b444a0922203ab90c2fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f83fcbdc88e4d679690f2d7ac48881c"
I20260812 01:11:20.386102 23828 raft_consensus.cc:3037] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:20.386245 23330 raft_consensus.cc:3037] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:20.389510 23828 raft_consensus.cc:2454] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb02a6b6b444a0922203ab90c2fba7 in term 1.
I20260812 01:11:20.389760 23330 raft_consensus.cc:2454] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eb02a6b6b444a0922203ab90c2fba7 in term 1.
I20260812 01:11:20.390154 23450 leader_election.cc:302] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eb02a6b6b444a0922203ab90c2fba7, 6ce75f6c019f43bf9112091974418250; no voters:
I20260812 01:11:20.391664 23971 raft_consensus.cc:2781] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:20.391850 23971 raft_consensus.cc:686] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 LEADER]: Becoming Leader. State: Replica: 28eb02a6b6b444a0922203ab90c2fba7, State: Running, Role: LEADER
I20260812 01:11:20.392163 23971 consensus_queue.cc:237] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } }
I20260812 01:11:20.394409 24042 tablet_replica.cc:439] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: TxnStatusTablet state changed. Reason: New leader 28eb02a6b6b444a0922203ab90c2fba7. Latest consensus state: current_term: 1 leader_uuid: "28eb02a6b6b444a0922203ab90c2fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } }
I20260812 01:11:20.395144 24042 tablet_replica.cc:442] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: This TxnStatusTablet replica's current role is: LEADER
I20260812 01:11:20.396171 24044 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 01:11:20.395860 23035 catalog_manager.cc:5495] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28eb02a6b6b444a0922203ab90c2fba7 (127.20.151.2). New cstate: current_term: 1 leader_uuid: "28eb02a6b6b444a0922203ab90c2fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } health_report { overall_health: UNKNOWN } } }
I20260812 01:11:20.396510 24044 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 01:11:20.393970 23971 consensus_queue.cc:1035] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260812 01:11:20.458617 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.450387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.20.151.2:40459)
0812 01:11:20.451947 (+ 1560us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:20.451952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:20.451972 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:11:20.452060 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:20.452064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:20.452153 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:20.452279 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:20.452808 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.453825 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.455859 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.456384 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.456481 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.457182 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:20.457805 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:20.457909 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:20.458100 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 01:11:20.458172 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1443,"thread_start_us":1258,"threads_started":1}
I20260812 01:11:20.458710 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.450643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:40459 (local address 127.0.0.1:48390)
0812 01:11:20.451117 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:20.451143 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:20.451351 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:20.452365 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:20.452370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:20.452397 (+ 27us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 01:11:20.452647 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:20.452663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.453979 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.453986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:20.455711 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:20.455728 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.456596 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.456605 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:20.456886 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.456970 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 01:11:20.457919 (+ 949us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 01:11:20.457953 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 01:11:20.458048 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 01:11:20.458334 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260812 01:11:20.464027 23330 raft_consensus.cc:1261] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c [term 1 FOLLOWER]: Refusing update from remote peer 28eb02a6b6b444a0922203ab90c2fba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:20.464012 23828 raft_consensus.cc:1261] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250 [term 1 FOLLOWER]: Refusing update from remote peer 28eb02a6b6b444a0922203ab90c2fba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:20.464839 23971 consensus_queue.cc:1035] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:20.465173 24020 consensus_queue.cc:1035] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:20.468403 23977 tablet_replica.cc:439] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250: TxnStatusTablet state changed. Reason: New leader 28eb02a6b6b444a0922203ab90c2fba7. Latest consensus state: current_term: 1 leader_uuid: "28eb02a6b6b444a0922203ab90c2fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } }
I20260812 01:11:20.469239 23977 tablet_replica.cc:442] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:20.469775 23968 tablet_replica.cc:439] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c: TxnStatusTablet state changed. Reason: New leader 28eb02a6b6b444a0922203ab90c2fba7. Latest consensus state: current_term: 1 leader_uuid: "28eb02a6b6b444a0922203ab90c2fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } }
I20260812 01:11:20.470168 23968 tablet_replica.cc:442] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:20.472863 24041 mvcc.cc:205] Tried to move back new op lower bound from 7317492041572638720 to 7317492041290006528. Current Snapshot: MvccSnapshot[applied={T|T < 7317492041572638720}]
I20260812 01:11:20.473770 24020 tablet_replica.cc:439] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28eb02a6b6b444a0922203ab90c2fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } }
I20260812 01:11:20.475349 23968 tablet_replica.cc:439] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28eb02a6b6b444a0922203ab90c2fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } }
I20260812 01:11:20.475598 23968 tablet_replica.cc:442] T e0945084ec4847138ef3dbea029b667d P 3f83fcbdc88e4d679690f2d7ac48881c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:20.476701 24020 tablet_replica.cc:442] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: This TxnStatusTablet replica's current role is: LEADER
I20260812 01:11:20.479388 24042 tablet_replica.cc:439] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28eb02a6b6b444a0922203ab90c2fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } }
I20260812 01:11:20.480818 24042 tablet_replica.cc:442] T e0945084ec4847138ef3dbea029b667d P 28eb02a6b6b444a0922203ab90c2fba7: This TxnStatusTablet replica's current role is: LEADER
I20260812 01:11:20.485271 23977 tablet_replica.cc:439] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28eb02a6b6b444a0922203ab90c2fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f83fcbdc88e4d679690f2d7ac48881c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 40373 } } peers { permanent_uuid: "28eb02a6b6b444a0922203ab90c2fba7" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40459 } } peers { permanent_uuid: "6ce75f6c019f43bf9112091974418250" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40645 } } }
I20260812 01:11:20.485574 23977 tablet_replica.cc:442] T e0945084ec4847138ef3dbea029b667d P 6ce75f6c019f43bf9112091974418250: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:20.505414 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.497072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48402 (local address 127.20.151.2:40459)
0812 01:11:20.497301 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:20.497309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:20.498575 (+ 1266us) server_negotiation.cc:407] Connection header received
0812 01:11:20.498896 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:20.498903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:20.498998 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:20.499099 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:20.499870 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.500990 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.503243 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.503837 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.503933 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.504513 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:20.504620 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:20.504683 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:20.504897 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 01:11:20.504980 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 01:11:20.526057 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.517748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:34977 (local address 127.20.151.2:40459)
0812 01:11:20.517970 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:20.517977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:20.519410 (+ 1433us) server_negotiation.cc:407] Connection header received
0812 01:11:20.519627 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:20.519631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:20.519736 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:20.519814 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:20.520505 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.521675 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.523712 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.524328 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.524440 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.525136 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:20.525246 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:20.525317 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:20.525447 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 01:11:20.525531 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 01:11:20.526083 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.518013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:40459 (local address 127.20.151.2:34977)
0812 01:11:20.519312 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:20.519333 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:20.519491 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:20.519841 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:20.519845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:20.519856 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 01:11:20.520363 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:20.520374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.521819 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.521825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:20.523584 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:20.523597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.524530 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.524536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:20.524873 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.524940 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 01:11:20.525338 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 01:11:20.525348 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 01:11:20.525423 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 01:11:20.525612 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"thread_start_us":765,"threads_started":1}
I20260812 01:11:20.600181 23034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:20.600502 23034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:20.604174 23034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2418963af3a34a7ca5e092ac2dc43138: Sending DeleteTablet for 3 replicas of tablet 088b68e106ec47ee8925da7ca44eab03
I20260812 01:11:20.605144 23294 tablet_service.cc:1514] Processing DeleteTablet for tablet 088b68e106ec47ee8925da7ca44eab03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:20 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:32788
I20260812 01:11:20.605243 23549 tablet_service.cc:1514] Processing DeleteTablet for tablet 088b68e106ec47ee8925da7ca44eab03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:20 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:48388
I20260812 01:11:20.605456 23796 tablet_service.cc:1514] Processing DeleteTablet for tablet 088b68e106ec47ee8925da7ca44eab03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:20 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:51676
I20260812 01:11:20.620996 24081 tablet_replica.cc:330] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c: stopping tablet replica
I20260812 01:11:20.621064 24082 tablet_replica.cc:330] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250: stopping tablet replica
I20260812 01:11:20.621274 24082 raft_consensus.cc:2229] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:20.621351 24081 raft_consensus.cc:2229] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:20.621728 24082 raft_consensus.cc:2258] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:20.622291 24081 raft_consensus.cc:2258] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:20.622998 24083 tablet_replica.cc:330] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7: stopping tablet replica
I20260812 01:11:20.623324 24082 ts_tablet_manager.cc:1905] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:20.623710 24083 raft_consensus.cc:2229] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:11:20.627116 24083 raft_consensus.cc:2258] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:20.628360 24083 ts_tablet_manager.cc:1905] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:20.629894 24081 ts_tablet_manager.cc:1905] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:20.641453 24082 ts_tablet_manager.cc:1918] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:11:20.641745 24082 log.cc:1196] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/088b68e106ec47ee8925da7ca44eab03
I20260812 01:11:20.642422 24082 ts_tablet_manager.cc:1939] T 088b68e106ec47ee8925da7ca44eab03 P 6ce75f6c019f43bf9112091974418250: Deleting consensus metadata
I20260812 01:11:20.643896 23011 catalog_manager.cc:4841] TS 6ce75f6c019f43bf9112091974418250 (127.20.151.3:40645): tablet 088b68e106ec47ee8925da7ca44eab03 (table test-table [id=3f572b0eaf534dd592b86972e4f23049]) successfully deleted
I20260812 01:11:20.644376 24083 ts_tablet_manager.cc:1918] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:11:20.644901 24083 log.cc:1196] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/088b68e106ec47ee8925da7ca44eab03
I20260812 01:11:20.645486 24083 ts_tablet_manager.cc:1939] T 088b68e106ec47ee8925da7ca44eab03 P 28eb02a6b6b444a0922203ab90c2fba7: Deleting consensus metadata
I20260812 01:11:20.646222 24081 ts_tablet_manager.cc:1918] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:11:20.646492 24081 log.cc:1196] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/088b68e106ec47ee8925da7ca44eab03
I20260812 01:11:20.647166 24081 ts_tablet_manager.cc:1939] T 088b68e106ec47ee8925da7ca44eab03 P 3f83fcbdc88e4d679690f2d7ac48881c: Deleting consensus metadata
I20260812 01:11:20.648377 23014 catalog_manager.cc:4841] TS 3f83fcbdc88e4d679690f2d7ac48881c (127.20.151.1:40373): tablet 088b68e106ec47ee8925da7ca44eab03 (table test-table [id=3f572b0eaf534dd592b86972e4f23049]) successfully deleted
I20260812 01:11:20.649022 23014 catalog_manager.cc:4841] TS 28eb02a6b6b444a0922203ab90c2fba7 (127.20.151.2:40459): tablet 088b68e106ec47ee8925da7ca44eab03 (table test-table [id=3f572b0eaf534dd592b86972e4f23049]) successfully deleted
Aug 12 01:11:20 dist-test-slave-21b9 krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:20.665300 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.623943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.20.151.1:40373)
0812 01:11:20.638284 (+ 14341us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:20.638290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:20.645276 (+ 6986us) server_negotiation.cc:407] Connection header received
0812 01:11:20.645620 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:20.645626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:20.645711 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:20.645791 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:20.648957 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.650032 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.651627 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.652101 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.652218 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.656873 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:20.656894 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:20.656916 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:20.656959 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:20.661568 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.662466 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.662477 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.662481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.662673 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.663098 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.663103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.663107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.663734 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:20.664131 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:20.664399 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 01:11:20.664769 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14174,"thread_start_us":14026,"threads_started":1}
W20260812 01:11:20.666179 23410 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32792
I20260812 01:11:20.687623 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.669473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.20.151.62:44023)
0812 01:11:20.671384 (+ 1911us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:20.671390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:20.671409 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:11:20.671468 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:20.671473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:20.671545 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:20.671637 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:20.674127 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.675091 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.677041 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.677541 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.677646 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.680976 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:20.680981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:20.680984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:20.681016 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:20.684288 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.685131 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.685134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.685136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.685355 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.685848 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.685853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.685855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.686242 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:20.686504 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:20.687087 (+ 583us) server_negotiation.cc:299] Negotiation successful
0812 01:11:20.687322 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1770,"thread_start_us":1594,"threads_started":1}
W20260812 01:11:20.688452 23034 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42764
I20260812 01:11:20.691555 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 23172
I20260812 01:11:20.705862 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.692415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32806 (local address 127.20.151.1:40373)
0812 01:11:20.692624 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:20.692630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:20.692921 (+ 291us) server_negotiation.cc:407] Connection header received
0812 01:11:20.693104 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:20.693108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:20.693176 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:20.693260 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:20.694479 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.695195 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.696375 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.696947 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.697066 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.699624 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:20.699629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:20.699632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:20.699666 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:20.702963 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.703711 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.703716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.703718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.703951 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.704352 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.704356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.704358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.704699 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:20.705075 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:20.705290 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 01:11:20.705532 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 01:11:20.706415 23410 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32806
W20260812 01:11:20.707119 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 23172: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:11:20.707330 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 23172
I20260812 01:11:20.738595 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 23426
Aug 12 01:11:20 dist-test-slave-21b9 krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:11:20.757843 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.739463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.20.151.2:40459)
0812 01:11:20.739804 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:20.739808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:20.739831 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:11:20.739891 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:20.739895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:20.740007 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:20.740096 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:20.741664 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.742533 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.744706 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.745481 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.745596 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.751159 (+ 5563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:20.751178 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:20.751196 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:20.751236 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:20.754720 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.755679 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.755691 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.755693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.755825 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.756245 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.756248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.756250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.756696 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:20.756978 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:20.757312 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 01:11:20.757538 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 01:11:20.758741 23666 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48416
W20260812 01:11:20.759473 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 23426: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:11:20.759661 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 23426
I20260812 01:11:20.797354 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 23680
Aug 12 01:11:20 dist-test-slave-21b9 krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:11:20.812748 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:20.798141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.20.151.3:40645)
0812 01:11:20.798654 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:20.798659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:20.798674 (+ 15us) server_negotiation.cc:407] Connection header received
0812 01:11:20.798729 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:20.798733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:20.798799 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:20.798893 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:20.800297 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.801065 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.802339 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:20.802823 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:20.802921 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:20.807107 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:20.807126 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:20.807149 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:20.807184 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:20.810000 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.810665 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.810671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.810674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.810807 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:20.811189 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:20.811192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:20.811193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:20.811706 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:20.811958 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:20.812179 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 01:11:20.812381 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":273,"threads_started":1}
W20260812 01:11:20.813512 23915 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51700
W20260812 01:11:20.814167 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 23680: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:11:20.814320 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 23680
I20260812 01:11:20.842602 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 22990
I20260812 01:11:20.843860 23103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:20.964286 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 22990
2026-08-12T01:11:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5273 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:11:21 dist-test-slave-21b9 krb5kdc[24111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:11:21 dist-test-slave-21b9 krb5kdc[24111](info): set up 2 sockets
Aug 12 01:11:21 dist-test-slave-21b9 krb5kdc[24111](info): commencing operation
krb5kdc: starting...
W20260812 01:11:22.225365 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:11:22 dist-test-slave-21b9 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:11:22.450981 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46757
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:46757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:22.795904 24145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:22.796243 24145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:22.796388 24145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:22.803758 24145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:22.803877 24145 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 01:11:22.803951 24145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:22.804041 24145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:22.804111 24145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:22.804198 24145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:11:22.814232 24145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:46757
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:22.815589 24145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:22.816922 24145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:22.828693 24158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:22.828575 24152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:22.828792 24151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:22.829862 24145 server_base.cc:1034] running on GCE node
I20260812 01:11:22.830685 24145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:22.832244 24145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:22.833487 24145 hybrid_clock.cc:648] HybridClock initialized: now 1786497082833404 us; error 52 us; skew 500 ppm
I20260812 01:11:22.833860 24145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:22 dist-test-slave-21b9 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497082, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:22.837319 24145 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:11:22.839003 24145 webserver.cc:460] Webserver started at http://127.20.151.62:38749/ using document root <none> and password file <none>
I20260812 01:11:22.839799 24145 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:22.839901 24145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:22.840247 24145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:22.843492 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "a58848daa399457897c15a91f6eea8b4"
format_stamp: "Formatted at 2026-08-12 01:11:22 on dist-test-slave-21b9"
server_key: "7792ba2a63dfb16d914335d7ebaf2831"
server_key_iv: "58ab4d02129c3a95769cf2d7f1d89ef0"
server_key_version: "encryptionkey@0"
I20260812 01:11:22.844578 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "a58848daa399457897c15a91f6eea8b4"
format_stamp: "Formatted at 2026-08-12 01:11:22 on dist-test-slave-21b9"
server_key: "7792ba2a63dfb16d914335d7ebaf2831"
server_key_iv: "58ab4d02129c3a95769cf2d7f1d89ef0"
server_key_version: "encryptionkey@0"
I20260812 01:11:22.849980 24145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 01:11:22.853889 24167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:22.855459 24145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:11:22.855646 24145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "a58848daa399457897c15a91f6eea8b4"
format_stamp: "Formatted at 2026-08-12 01:11:22 on dist-test-slave-21b9"
server_key: "7792ba2a63dfb16d914335d7ebaf2831"
server_key_iv: "58ab4d02129c3a95769cf2d7f1d89ef0"
server_key_version: "encryptionkey@0"
I20260812 01:11:22.855882 24145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:22.871953 24145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:22.876612 24145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:22.876895 24145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:22.897527 24145 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:46757
I20260812 01:11:22.897538 24263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:46757 every 8 connection(s)
I20260812 01:11:22.898895 24145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:22.902721 24264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:22.904505 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 24145
I20260812 01:11:22.904698 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:22.905336 21084 external_mini_cluster.cc:1428] Setting key 5db8900049f59b47bb691ffdc185021b
I20260812 01:11:22.914870 24264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4: Bootstrap starting.
I20260812 01:11:22.918731 24264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:22.920024 24264 log.cc:824] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4: Log is configured to *not* fsync() on all Append() calls
Aug 12 01:11:22 dist-test-slave-21b9 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:22.923010 24264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4: No bootstrap required, opened a new log
I20260812 01:11:22.928578 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:22.907317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.20.151.62:46757)
0812 01:11:22.908172 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:22.908183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:22.908231 (+ 48us) server_negotiation.cc:407] Connection header received
0812 01:11:22.909961 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:22.909984 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:22.910350 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:22.910663 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:22.912501 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:22.913824 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:22.915570 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:22.916295 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:22.916447 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:22.921269 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:22.921280 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:22.921294 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:22.921334 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:22.924437 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:22.925272 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:22.925278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:22.925285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:22.925462 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:22.925850 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:22.925853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:22.925855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:22.926417 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:22.926684 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:22.927081 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 01:11:22.927487 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":392,"threads_started":1}
I20260812 01:11:22.932147 24264 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58848daa399457897c15a91f6eea8b4" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46757 } }
I20260812 01:11:22.932498 24264 raft_consensus.cc:374] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:22.932595 24264 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58848daa399457897c15a91f6eea8b4, State: Initialized, Role: FOLLOWER
I20260812 01:11:22.933140 24264 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58848daa399457897c15a91f6eea8b4" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46757 } }
I20260812 01:11:22.933370 24264 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:22.933507 24264 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:22.933725 24264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:22.936091 24264 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58848daa399457897c15a91f6eea8b4" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46757 } }
I20260812 01:11:22.936594 24264 leader_election.cc:302] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58848daa399457897c15a91f6eea8b4; no voters:
I20260812 01:11:22.937160 24264 leader_election.cc:288] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:22.937297 24272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:22.938068 24272 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [term 1 LEADER]: Becoming Leader. State: Replica: a58848daa399457897c15a91f6eea8b4, State: Running, Role: LEADER
I20260812 01:11:22.938582 24272 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58848daa399457897c15a91f6eea8b4" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46757 } }
I20260812 01:11:22.939477 24264 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:22.941567 24273 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a58848daa399457897c15a91f6eea8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58848daa399457897c15a91f6eea8b4" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46757 } } }
I20260812 01:11:22.942047 24273 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:22.942288 24274 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58848daa399457897c15a91f6eea8b4. Latest consensus state: current_term: 1 leader_uuid: "a58848daa399457897c15a91f6eea8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58848daa399457897c15a91f6eea8b4" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46757 } } }
I20260812 01:11:22.942629 24274 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:22.944339 24281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:22.948904 24281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:22.957392 24281 catalog_manager.cc:1347] Generated new cluster ID: 2c2bc3b743c24d60a0a4a0db0b0f2185
I20260812 01:11:22.957540 24281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:22.965773 24281 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:22.967244 24281 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:22.977028 24281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4: Generated new TSK 0
I20260812 01:11:22.977895 24281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:11:23.031037 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46757
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:23.383882 24303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:23.384188 24303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:23.384354 24303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:23.392006 24303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:23.392158 24303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:23.392298 24303 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 01:11:23.392375 24303 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 01:11:23.392503 24303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:11:23.403043 24303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.151.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:23.404395 24303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:23.405774 24303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:23.418984 24319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:23.419081 24314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:23.419085 24312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:23.424829 24303 server_base.cc:1034] running on GCE node
I20260812 01:11:23.425485 24303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:23.426466 24303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:23.427683 24303 hybrid_clock.cc:648] HybridClock initialized: now 1786497083427632 us; error 49 us; skew 500 ppm
I20260812 01:11:23.427996 24303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:23 dist-test-slave-21b9 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497083, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:23.431319 24303 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:11:23.432802 24303 webserver.cc:460] Webserver started at http://127.20.151.1:33149/ using document root <none> and password file <none>
I20260812 01:11:23.433506 24303 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:23.433647 24303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:23.433954 24303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:23.436908 24303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "77ce7bb678944d939eb6f722b518f605"
format_stamp: "Formatted at 2026-08-12 01:11:23 on dist-test-slave-21b9"
server_key: "df8c5fd689028afae4482b9fa29f9f7c"
server_key_iv: "44de6b2b43446265af396d273629b96e"
server_key_version: "encryptionkey@0"
I20260812 01:11:23.437646 24303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "77ce7bb678944d939eb6f722b518f605"
format_stamp: "Formatted at 2026-08-12 01:11:23 on dist-test-slave-21b9"
server_key: "df8c5fd689028afae4482b9fa29f9f7c"
server_key_iv: "44de6b2b43446265af396d273629b96e"
server_key_version: "encryptionkey@0"
I20260812 01:11:23.442945 24303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:11:23.446569 24331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:23.447907 24303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 01:11:23.448073 24303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "77ce7bb678944d939eb6f722b518f605"
format_stamp: "Formatted at 2026-08-12 01:11:23 on dist-test-slave-21b9"
server_key: "df8c5fd689028afae4482b9fa29f9f7c"
server_key_iv: "44de6b2b43446265af396d273629b96e"
server_key_version: "encryptionkey@0"
I20260812 01:11:23.448272 24303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:23.461737 24303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:23.465830 24303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:23.466104 24303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:23.468852 24303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:23.468956 24303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:23.469069 24303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:23.469183 24303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 01:11:23 dist-test-slave-21b9 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497083, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:23.492565 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:23.469445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:38709 (local address 127.20.151.62:46757)
0812 01:11:23.470205 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:23.470211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:23.471206 (+ 995us) server_negotiation.cc:407] Connection header received
0812 01:11:23.473405 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:23.473412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:23.473505 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:23.473644 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:23.475852 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:23.477086 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:23.479982 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:23.480549 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:23.480683 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:23.485761 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:23.485768 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:23.485773 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:23.485816 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:23.488708 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:23.489617 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:23.489621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:23.489624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:23.489742 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:23.490284 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:23.490290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:23.490293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:23.490659 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:23.490801 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:23.491869 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0812 01:11:23.492141 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":484,"threads_started":1}
I20260812 01:11:23.493427 24354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:23.470116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46757 (local address 127.20.151.1:38709)
0812 01:11:23.471023 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:23.471081 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:23.473107 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:23.473899 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:23.473913 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:23.474688 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:23.475633 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:23.475663 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:23.477294 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:23.477302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:23.479806 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:23.479825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:23.480988 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:23.480998 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:23.481280 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:23.482029 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:23.482068 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:23.485461 (+ 3393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:23.488841 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:23.488853 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:23.488871 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:23.489498 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:23.489854 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:23.489860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:23.489864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:23.490146 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:23.490805 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:23.490815 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:23.491387 (+ 572us) client_negotiation.cc:769] Sending connection context
0812 01:11:23.491730 (+ 343us) client_negotiation.cc:240] Negotiation successful
0812 01:11:23.492207 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":394,"threads_started":1}
I20260812 01:11:23.514842 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:23.499178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:38793 (local address 127.20.151.62:46757)
0812 01:11:23.499404 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:23.499410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:23.500245 (+ 835us) server_negotiation.cc:407] Connection header received
0812 01:11:23.500439 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:23.500445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:23.500539 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:23.500651 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:23.502371 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:23.503491 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:23.505979 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:23.506380 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:23.506550 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:23.508980 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:23.508985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:23.508988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:23.509021 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:23.511891 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:23.512743 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:23.512748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:23.512752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:23.512904 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:23.513370 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:23.513375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:23.513378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:23.513682 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:23.513819 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:23.514237 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 01:11:23.514469 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 01:11:23.514925 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:23.499188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46757 (local address 127.20.151.1:38793)
0812 01:11:23.500128 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:23.500152 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:23.500296 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:23.500844 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:23.500850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:23.501547 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:23.502158 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:23.502175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:23.503689 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:23.503699 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:23.505777 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:23.505797 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:23.506686 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:23.506697 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:23.506870 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:23.507661 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:23.507685 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:23.508732 (+ 1047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:23.512063 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:23.512069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:23.512073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:23.512629 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:23.513041 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:23.513047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:23.513051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:23.513244 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:23.513804 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:23.513810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:23.513997 (+ 187us) client_negotiation.cc:769] Sending connection context
0812 01:11:23.514117 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 01:11:23.514452 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":596,"threads_started":1}
I20260812 01:11:23.518841 24303 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:41977
I20260812 01:11:23.518989 24555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:41977 every 8 connection(s)
I20260812 01:11:23.520114 24303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:11:23.520680 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 24303
I20260812 01:11:23.520895 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:11:23.521703 21084 external_mini_cluster.cc:1428] Setting key f5a675fca328a0d0ce6201b588b5b556
I20260812 01:11:23.522470 24556 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46757
I20260812 01:11:23.522670 24556 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:23.523465 24556 heartbeater.cc:511] Master 127.20.151.62:46757 requested a full tablet report, sending...
I20260812 01:11:23.525666 24191 ts_manager.cc:194] Registered new tserver with Master: 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977)
I20260812 01:11:23.527544 24191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:38709
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:11:23.574100 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46757
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:23.914222 24562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:23.914508 24562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:23.914644 24562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:23.922044 24562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:23.922163 24562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:23.922286 24562 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 01:11:23.922349 24562 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 01:11:23.922447 24562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:11:23.932474 24562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.151.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:23.933782 24562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:23.935132 24562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:23.948546 24573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:23.948802 24578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:23.949326 24572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:23.949626 24562 server_base.cc:1034] running on GCE node
I20260812 01:11:23.950265 24562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:23.951160 24562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:23.952356 24562 hybrid_clock.cc:648] HybridClock initialized: now 1786497083952296 us; error 80 us; skew 500 ppm
I20260812 01:11:23.952679 24562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:23 dist-test-slave-21b9 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497083, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:23.955775 24562 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:11:23.957178 24562 webserver.cc:460] Webserver started at http://127.20.151.2:42371/ using document root <none> and password file <none>
I20260812 01:11:23.957854 24562 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:23.957974 24562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:23.958276 24562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:23.961320 24562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "dfd7cf253ae04d4789dcf4a626c9a175"
format_stamp: "Formatted at 2026-08-12 01:11:23 on dist-test-slave-21b9"
server_key: "8a6bbde962182a7c9ce1ca1f6d154db8"
server_key_iv: "66a7e6800ca3f60163554bc9708678bc"
server_key_version: "encryptionkey@0"
I20260812 01:11:23.962049 24562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "dfd7cf253ae04d4789dcf4a626c9a175"
format_stamp: "Formatted at 2026-08-12 01:11:23 on dist-test-slave-21b9"
server_key: "8a6bbde962182a7c9ce1ca1f6d154db8"
server_key_iv: "66a7e6800ca3f60163554bc9708678bc"
server_key_version: "encryptionkey@0"
I20260812 01:11:23.967108 24562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 01:11:23.970549 24589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:23.971926 24562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:11:23.972091 24562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "dfd7cf253ae04d4789dcf4a626c9a175"
format_stamp: "Formatted at 2026-08-12 01:11:23 on dist-test-slave-21b9"
server_key: "8a6bbde962182a7c9ce1ca1f6d154db8"
server_key_iv: "66a7e6800ca3f60163554bc9708678bc"
server_key_version: "encryptionkey@0"
I20260812 01:11:23.972276 24562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:23.998117 24562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:24.002252 24562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:24.002532 24562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:24.005363 24562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:24.005470 24562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:24.005582 24562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:24.005682 24562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 01:11:24 dist-test-slave-21b9 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497083, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:24.030109 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.006070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:46241 (local address 127.20.151.62:46757)
0812 01:11:24.006298 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:24.006305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:24.007826 (+ 1521us) server_negotiation.cc:407] Connection header received
0812 01:11:24.010316 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:24.010324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:24.010425 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:24.010503 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:24.012866 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.013945 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.016517 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.017195 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.017295 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.022281 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:24.022289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:24.022294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:24.022339 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:24.025993 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.026958 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.026963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.026968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.027118 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.027788 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.027793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.027796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.028206 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:24.028372 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:24.029414 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0812 01:11:24.029689 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 01:11:24.031406 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.006662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46757 (local address 127.20.151.2:46241)
0812 01:11:24.007606 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:24.007666 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:24.009893 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:24.011073 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:24.011084 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:24.011847 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:24.012658 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.012684 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.014169 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.014174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.016322 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.016339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.017700 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.017707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.017984 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.018806 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:24.018837 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:24.022005 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:24.026161 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:24.026189 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:24.026208 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:24.026848 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:24.027242 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:24.027247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:24.027250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:24.027627 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:24.028354 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:24.028363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:24.028870 (+ 507us) client_negotiation.cc:769] Sending connection context
0812 01:11:24.029246 (+ 376us) client_negotiation.cc:240] Negotiation successful
0812 01:11:24.029860 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":376,"threads_started":1}
I20260812 01:11:24.053766 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.038322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:49427 (local address 127.20.151.62:46757)
0812 01:11:24.038527 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:24.038534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:24.039149 (+ 615us) server_negotiation.cc:407] Connection header received
0812 01:11:24.039374 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:24.039380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:24.039461 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:24.039564 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:24.040584 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.041727 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.043875 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.044612 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.044760 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.046855 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:24.046863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:24.046869 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:24.046914 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:24.050425 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.051393 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.051399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.051403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.051561 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.052031 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.052036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.052040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.052515 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:24.052705 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:24.053309 (+ 604us) server_negotiation.cc:299] Negotiation successful
0812 01:11:24.053531 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 01:11:24.054713 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.038317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46757 (local address 127.20.151.2:49427)
0812 01:11:24.039021 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:24.039041 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:24.039184 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:24.039605 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:24.039608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:24.040105 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:24.040437 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.040466 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.041910 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.041916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.043696 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.043715 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.044841 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.044846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.044968 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.045828 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:24.045847 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:24.046640 (+ 793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:24.050605 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:24.050609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:24.050612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:24.051212 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:24.051719 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:24.051724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:24.051727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:24.051914 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:24.052775 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:24.052780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:24.053026 (+ 246us) client_negotiation.cc:769] Sending connection context
0812 01:11:24.053205 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 01:11:24.054212 (+ 1007us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":389,"threads_started":1}
I20260812 01:11:24.070071 24562 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:43355
I20260812 01:11:24.070113 24787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:43355 every 8 connection(s)
I20260812 01:11:24.071427 24562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:11:24.073581 24789 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46757
I20260812 01:11:24.073827 24789 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:24.074517 24789 heartbeater.cc:511] Master 127.20.151.62:46757 requested a full tablet report, sending...
I20260812 01:11:24.076093 24191 ts_manager.cc:194] Registered new tserver with Master: dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2:43355)
I20260812 01:11:24.077045 24191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:46241
I20260812 01:11:24.078083 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 24562
I20260812 01:11:24.078279 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:11:24.078847 21084 external_mini_cluster.cc:1428] Setting key a04197c348320056b6cbe035473f6792
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:11:24.137888 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46757
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:24.487076 24793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:24.487380 24793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:24.487533 24793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:24.494769 24793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:24.494892 24793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:24.495026 24793 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 01:11:24.495090 24793 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 01:11:24.495198 24793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:11:24.505165 24793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.151.62:46757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:24.506532 24793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:24.507989 24793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:24.522776 24802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:24.522748 24807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:24.524055 24809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:24.525627 24793 server_base.cc:1034] running on GCE node
I20260812 01:11:24.526212 24793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:24.527112 24793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:24.528312 24793 hybrid_clock.cc:648] HybridClock initialized: now 1786497084528253 us; error 44 us; skew 500 ppm
I20260812 01:11:24.528620 24793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:11:24.530982 24556 heartbeater.cc:503] Master 127.20.151.62:46757 was elected leader, sending a full tablet report...
Aug 12 01:11:24 dist-test-slave-21b9 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497084, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:24.534768 24793 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:11:24.536268 24793 webserver.cc:460] Webserver started at http://127.20.151.3:39911/ using document root <none> and password file <none>
I20260812 01:11:24.536968 24793 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:24.537096 24793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:24.537400 24793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:24.540382 24793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "c89c70a93e614d6c9780683b8dc5be41"
format_stamp: "Formatted at 2026-08-12 01:11:24 on dist-test-slave-21b9"
server_key: "87467d00942f78c218c01d522fcb649f"
server_key_iv: "c6a718e1f2f65642927561c447896105"
server_key_version: "encryptionkey@0"
I20260812 01:11:24.541162 24793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "c89c70a93e614d6c9780683b8dc5be41"
format_stamp: "Formatted at 2026-08-12 01:11:24 on dist-test-slave-21b9"
server_key: "87467d00942f78c218c01d522fcb649f"
server_key_iv: "c6a718e1f2f65642927561c447896105"
server_key_version: "encryptionkey@0"
I20260812 01:11:24.546474 24793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:11:24.550241 24825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:24.551620 24793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:11:24.551769 24793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "c89c70a93e614d6c9780683b8dc5be41"
format_stamp: "Formatted at 2026-08-12 01:11:24 on dist-test-slave-21b9"
server_key: "87467d00942f78c218c01d522fcb649f"
server_key_iv: "c6a718e1f2f65642927561c447896105"
server_key_version: "encryptionkey@0"
I20260812 01:11:24.551959 24793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:24.576462 24793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:24.580596 24793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:24.580878 24793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:24.583796 24793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:24.583922 24793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:24.584052 24793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:24.584160 24793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 01:11:24 dist-test-slave-21b9 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497084, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:24.607549 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.584249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:45987 (local address 127.20.151.62:46757)
0812 01:11:24.585079 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:24.585083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:24.586024 (+ 941us) server_negotiation.cc:407] Connection header received
0812 01:11:24.588143 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:24.588150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:24.588234 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:24.588332 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:24.590545 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.591564 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.594437 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.594963 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.595097 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.600128 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:24.600136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:24.600141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:24.600191 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:24.603768 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.604757 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.604763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.604766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.604947 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.605407 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.605412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.605416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.605844 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:24.606024 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:24.606806 (+ 782us) server_negotiation.cc:299] Negotiation successful
0812 01:11:24.607086 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":602,"threads_started":1}
I20260812 01:11:24.608469 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.584985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46757 (local address 127.20.151.3:45987)
0812 01:11:24.585864 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:24.585908 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:24.587858 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:24.588604 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:24.588613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:24.589408 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:24.590347 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.590368 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.591767 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.591774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.594269 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.594286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.595245 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.595253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.595605 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.596556 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:24.596595 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:24.599841 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:24.603906 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:24.603915 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:24.603938 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:24.604620 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:24.605075 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:24.605079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:24.605081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:24.605282 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:24.605957 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:24.605962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:24.606325 (+ 363us) client_negotiation.cc:769] Sending connection context
0812 01:11:24.606701 (+ 376us) client_negotiation.cc:240] Negotiation successful
0812 01:11:24.607121 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":376,"threads_started":1}
I20260812 01:11:24.628790 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.614731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:45151 (local address 127.20.151.62:46757)
0812 01:11:24.614908 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:24.614914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:24.615610 (+ 696us) server_negotiation.cc:407] Connection header received
0812 01:11:24.615783 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:24.615789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:24.615881 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:24.615978 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:24.617216 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.617893 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.619914 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.620437 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.620566 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.622888 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:24.622894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:24.622897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:24.622928 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:24.626066 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.626879 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.626885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.626889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.627029 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.627398 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.627403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.627406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.627682 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:24.627853 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:24.628361 (+ 508us) server_negotiation.cc:299] Negotiation successful
0812 01:11:24.628542 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 01:11:24.629161 24965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.614996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46757 (local address 127.20.151.3:45151)
0812 01:11:24.615515 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:24.615530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:24.615670 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:24.616020 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:24.616027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:24.616669 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:24.617074 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.617088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.618009 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.618014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.619768 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.619784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.620670 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.620682 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.620916 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.621679 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:24.621703 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:24.622702 (+ 999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:24.626197 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:24.626206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:24.626210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:24.626771 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:24.627134 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:24.627139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:24.627143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:24.627306 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:24.627818 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:24.627823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:24.628079 (+ 256us) client_negotiation.cc:769] Sending connection context
0812 01:11:24.628258 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 01:11:24.628634 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":275,"threads_started":1}
I20260812 01:11:24.632372 24793 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:43959
I20260812 01:11:24.632403 25033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:43959 every 8 connection(s)
I20260812 01:11:24.633848 24793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:11:24.635792 25037 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46757
I20260812 01:11:24.636009 25037 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:24.636646 25037 heartbeater.cc:511] Master 127.20.151.62:46757 requested a full tablet report, sending...
I20260812 01:11:24.637874 24191 ts_manager.cc:194] Registered new tserver with Master: c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3:43959)
I20260812 01:11:24.638653 24191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:45987
I20260812 01:11:24.643246 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 24793
I20260812 01:11:24.643424 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:11:24.643864 21084 external_mini_cluster.cc:1428] Setting key ad6c572abe0552e832ea377805e14eb5
I20260812 01:11:24.646602 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 01:11:24 dist-test-slave-21b9 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 01:11:24 dist-test-slave-21b9 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:24.701453 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.688171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.20.151.62:46757)
0812 01:11:24.688394 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:24.688401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:24.688622 (+ 221us) server_negotiation.cc:407] Connection header received
0812 01:11:24.688798 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:24.688802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:24.688876 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:24.688959 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:24.690315 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.691033 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.692275 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.692694 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.692786 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.696536 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:24.696542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:24.696545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:24.696580 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:24.699097 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.699690 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.699694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.699696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.699838 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.700258 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.700262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.700265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.700553 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:24.700761 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:24.700962 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 01:11:24.701141 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 01:11:24.705506 24191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:11:24.712692 24191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:11:24.737975 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.726350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59812 (local address 127.20.151.2:43355)
0812 01:11:24.729521 (+ 3171us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:24.729528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:24.729555 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:11:24.729683 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:24.729690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:24.729937 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:24.730141 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:24.730904 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.732373 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.735346 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.736399 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.736554 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.736793 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:24.737532 (+ 739us) server_negotiation.cc:299] Negotiation successful
0812 01:11:24.737696 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3018,"thread_start_us":2885,"threads_started":1}
I20260812 01:11:24.738857 25061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.725788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:43355 (local address 127.0.0.1:59812)
0812 01:11:24.728864 (+ 3076us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:24.728883 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:24.729008 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:24.730164 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:24.730172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:24.730203 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:24.730683 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.730704 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.732512 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.732522 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.735145 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.735169 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.736942 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.736950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.737196 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.737262 (+ 66us) client_negotiation.cc:769] Sending connection context
0812 01:11:24.738213 (+ 951us) client_negotiation.cc:240] Negotiation successful
0812 01:11:24.738409 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2937,"mutex_wait_us":4,"thread_start_us":574,"threads_started":1}
I20260812 01:11:24.739609 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.725600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:43959 (local address 127.0.0.1:54998)
0812 01:11:24.727471 (+ 1871us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:24.727528 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:24.727731 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:24.731885 (+ 4154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:24.731890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:24.731940 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:24.732395 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.732410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.734544 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.734548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.736785 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.736796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.738757 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.738764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.738939 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.738983 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 01:11:24.739073 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 01:11:24.739201 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1537,"mutex_wait_us":1,"spinlock_wait_cycles":3968,"thread_start_us":4884,"threads_started":1}
I20260812 01:11:24.739720 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.726573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.20.151.1:41977)
0812 01:11:24.728367 (+ 1794us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:24.728377 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:24.730904 (+ 2527us) server_negotiation.cc:407] Connection header received
0812 01:11:24.731211 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:24.731219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:24.731488 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:24.731647 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:24.733027 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.734202 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.736946 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.738422 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.738596 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.738808 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:24.739157 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 01:11:24.739360 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1603,"thread_start_us":1435,"threads_started":1}
I20260812 01:11:24.740509 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.726327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41977 (local address 127.0.0.1:34492)
0812 01:11:24.730787 (+ 4460us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:24.730804 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:24.731014 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:24.731900 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:24.731905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:24.731947 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:24.732807 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.732820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.734376 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.734379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.736749 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.736767 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.738689 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.738697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.738887 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.738932 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 01:11:24.740036 (+ 1104us) client_negotiation.cc:240] Negotiation successful
0812 01:11:24.740181 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4309,"mutex_wait_us":78,"spinlock_wait_cycles":1408,"thread_start_us":2240,"threads_started":1}
I20260812 01:11:24.740679 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.725821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.20.151.3:43959)
0812 01:11:24.731384 (+ 5563us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:24.731390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:24.731428 (+ 38us) server_negotiation.cc:407] Connection header received
0812 01:11:24.731511 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:24.731516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:24.731685 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:24.731879 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:24.732652 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.734388 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.736963 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.738511 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.739585 (+ 1074us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.740121 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:24.740190 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 01:11:24.740351 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5377,"thread_start_us":5174,"threads_started":1}
I20260812 01:11:24.743389 24683 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0905121a8d446c995b70077cb6220a (DEFAULT_TABLE table=test-table [id=da64ce5179c94a07a178da862de78297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:24.743966 24426 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0905121a8d446c995b70077cb6220a (DEFAULT_TABLE table=test-table [id=da64ce5179c94a07a178da862de78297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:24.744191 24909 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0905121a8d446c995b70077cb6220a (DEFAULT_TABLE table=test-table [id=da64ce5179c94a07a178da862de78297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:24.745029 24683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d0905121a8d446c995b70077cb6220a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:24.745234 24426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d0905121a8d446c995b70077cb6220a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:24.745626 24909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d0905121a8d446c995b70077cb6220a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:24.758244 25065 tablet_bootstrap.cc:492] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605: Bootstrap starting.
I20260812 01:11:24.762727 25065 tablet_bootstrap.cc:654] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:24.763933 25065 log.cc:824] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:24.766191 25068 tablet_bootstrap.cc:492] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: Bootstrap starting.
I20260812 01:11:24.766875 25065 tablet_bootstrap.cc:492] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605: No bootstrap required, opened a new log
I20260812 01:11:24.767158 25065 ts_tablet_manager.cc:1397] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260812 01:11:24.769224 25068 tablet_bootstrap.cc:654] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:24.770179 25071 tablet_bootstrap.cc:492] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: Bootstrap starting.
I20260812 01:11:24.770452 25068 log.cc:824] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:24.774390 25068 tablet_bootstrap.cc:492] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: No bootstrap required, opened a new log
I20260812 01:11:24.774755 25068 ts_tablet_manager.cc:1397] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260812 01:11:24.775111 25071 tablet_bootstrap.cc:654] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:24.776211 25065 raft_consensus.cc:348] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } }
I20260812 01:11:24.776597 25065 raft_consensus.cc:374] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:24.776690 25065 raft_consensus.cc:729] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ce7bb678944d939eb6f722b518f605, State: Initialized, Role: FOLLOWER
I20260812 01:11:24.776656 25071 log.cc:824] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:24.777330 25065 consensus_queue.cc:260] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } }
I20260812 01:11:24.779199 25065 ts_tablet_manager.cc:1428] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605: Time spent starting tablet: real 0.012s user 0.001s sys 0.011s
I20260812 01:11:24.780030 25071 tablet_bootstrap.cc:492] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: No bootstrap required, opened a new log
I20260812 01:11:24.780301 25071 ts_tablet_manager.cc:1397] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260812 01:11:24.785313 25068 raft_consensus.cc:348] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } }
I20260812 01:11:24.785791 25068 raft_consensus.cc:374] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:24.785913 25068 raft_consensus.cc:729] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7cf253ae04d4789dcf4a626c9a175, State: Initialized, Role: FOLLOWER
I20260812 01:11:24.786556 25068 consensus_queue.cc:260] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } }
I20260812 01:11:24.787798 24789 heartbeater.cc:503] Master 127.20.151.62:46757 was elected leader, sending a full tablet report...
I20260812 01:11:24.788352 25068 ts_tablet_manager.cc:1428] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: Time spent starting tablet: real 0.013s user 0.009s sys 0.006s
I20260812 01:11:24.789448 25071 raft_consensus.cc:348] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } }
I20260812 01:11:24.789889 25071 raft_consensus.cc:374] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:24.790014 25071 raft_consensus.cc:729] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89c70a93e614d6c9780683b8dc5be41, State: Initialized, Role: FOLLOWER
I20260812 01:11:24.790668 25071 consensus_queue.cc:260] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } }
I20260812 01:11:24.791785 25037 heartbeater.cc:503] Master 127.20.151.62:46757 was elected leader, sending a full tablet report...
I20260812 01:11:24.792474 25071 ts_tablet_manager.cc:1428] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
W20260812 01:11:24.823567 24790 tablet.cc:2369] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:24.879161 25077 raft_consensus.cc:482] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:11:24.879528 25077 raft_consensus.cc:504] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } }
I20260812 01:11:24.881314 25077 leader_election.cc:288] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3:43959), 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977)
W20260812 01:11:24.885459 25038 tablet.cc:2369] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:24.888587 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.881831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:37715 (local address 127.20.151.3:43959)
0812 01:11:24.882061 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:24.882068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:24.882731 (+ 663us) server_negotiation.cc:407] Connection header received
0812 01:11:24.882924 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:24.882928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:24.882988 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:24.883069 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:24.883782 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.884724 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.886871 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.887692 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.887842 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.887916 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:24.888216 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 01:11:24.888345 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 01:11:24.888717 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.881582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:43959 (local address 127.20.151.2:37715)
0812 01:11:24.882617 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:24.882633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:24.882785 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:24.883106 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:24.883110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:24.883134 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:24.883604 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.883623 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.884891 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.884896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.886715 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.886729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.887904 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.887909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.888056 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.888080 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 01:11:24.888164 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 01:11:24.888294 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":25,"thread_start_us":687,"threads_started":1}
I20260812 01:11:24.888828 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.881884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:35695 (local address 127.20.151.1:41977)
0812 01:11:24.882095 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:24.882115 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:24.882829 (+ 714us) server_negotiation.cc:407] Connection header received
0812 01:11:24.882980 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:24.882984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:24.883054 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:24.883130 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:24.883930 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.884767 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.887151 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.887967 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.888124 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.888195 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:24.888478 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 01:11:24.888610 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:11:24.889209 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.881772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41977 (local address 127.20.151.2:35695)
0812 01:11:24.882714 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:24.882728 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:24.882852 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:24.883160 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:24.883164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:24.883192 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:24.883738 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.883754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.884907 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.884912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.886999 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.887012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.888195 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.888201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.888337 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.888360 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 01:11:24.888447 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 01:11:24.888581 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":25,"thread_start_us":525,"threads_started":1}
I20260812 01:11:24.889936 24942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0905121a8d446c995b70077cb6220a" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89c70a93e614d6c9780683b8dc5be41" is_pre_election: true
I20260812 01:11:24.890195 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0905121a8d446c995b70077cb6220a" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ce7bb678944d939eb6f722b518f605" is_pre_election: true
I20260812 01:11:24.890587 24942 raft_consensus.cc:2454] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 0.
I20260812 01:11:24.890714 24463 raft_consensus.cc:2454] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 0.
I20260812 01:11:24.891374 24595 leader_election.cc:302] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c89c70a93e614d6c9780683b8dc5be41, dfd7cf253ae04d4789dcf4a626c9a175; no voters:
I20260812 01:11:24.891765 25077 raft_consensus.cc:2781] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:11:24.891902 25077 raft_consensus.cc:482] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:11:24.892004 25077 raft_consensus.cc:3037] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:24.894140 25077 raft_consensus.cc:504] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } }
I20260812 01:11:24.894730 25077 leader_election.cc:288] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 election: Requested vote from peers c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3:43959), 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977)
I20260812 01:11:24.895255 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0905121a8d446c995b70077cb6220a" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ce7bb678944d939eb6f722b518f605"
I20260812 01:11:24.895255 24942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0905121a8d446c995b70077cb6220a" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89c70a93e614d6c9780683b8dc5be41"
I20260812 01:11:24.895483 24463 raft_consensus.cc:3037] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:24.895484 24942 raft_consensus.cc:3037] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:24.898586 24463 raft_consensus.cc:2454] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 1.
I20260812 01:11:24.898586 24942 raft_consensus.cc:2454] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 1.
I20260812 01:11:24.898957 24595 leader_election.cc:302] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c89c70a93e614d6c9780683b8dc5be41, dfd7cf253ae04d4789dcf4a626c9a175; no voters:
I20260812 01:11:24.899255 25077 raft_consensus.cc:2781] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:24.899739 25077 raft_consensus.cc:686] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 LEADER]: Becoming Leader. State: Replica: dfd7cf253ae04d4789dcf4a626c9a175, State: Running, Role: LEADER
I20260812 01:11:24.900278 25077 consensus_queue.cc:237] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } }
I20260812 01:11:24.905527 24191 catalog_manager.cc:5495] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2). New cstate: current_term: 1 leader_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } health_report { overall_health: UNKNOWN } } }
Aug 12 01:11:24 dist-test-slave-21b9 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497082, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:24.959407 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.945873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46757 (local address 127.0.0.1:60310)
0812 01:11:24.946542 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:24.946558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:24.946719 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:24.947078 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:24.947082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:24.947877 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:24.948151 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.948160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.949283 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.949286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.950247 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:24.950258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.951084 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.951088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:24.951247 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.951892 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:24.951909 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:24.954037 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:24.956922 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:24.956933 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:24.956939 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:24.957422 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:24.957759 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:24.957762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:24.957765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:24.957898 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:24.958492 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:24.958498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:24.958668 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 01:11:24.958804 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 01:11:24.958998 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":341,"threads_started":1}
I20260812 01:11:24.959410 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:24.945834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.20.151.62:46757)
0812 01:11:24.946017 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:24.946021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:24.946682 (+ 661us) server_negotiation.cc:407] Connection header received
0812 01:11:24.946867 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:24.946872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:24.946946 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:24.947042 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:24.948310 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.949168 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.950426 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:24.950909 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:24.951012 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:24.954229 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:24.954234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:24.954238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:24.954269 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:24.956816 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.957550 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.957554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.957556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.957677 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:24.958020 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:24.958024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:24.958026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:24.958345 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:24.958517 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:24.958926 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 01:11:24.959120 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 01:11:24.964401 24191 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:60310:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 01:11:24.965428 24191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:11:24.972007 24426 tablet_service.cc:1467] Processing CreateTablet for tablet 45a254ad5ee047328b7d98dc3fb02396 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5aa32683c2f54112ac0da81d1ed31263]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 01:11:24.972047 24683 tablet_service.cc:1467] Processing CreateTablet for tablet 45a254ad5ee047328b7d98dc3fb02396 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5aa32683c2f54112ac0da81d1ed31263]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 01:11:24.972296 24909 tablet_service.cc:1467] Processing CreateTablet for tablet 45a254ad5ee047328b7d98dc3fb02396 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5aa32683c2f54112ac0da81d1ed31263]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 01:11:24.972944 24426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a254ad5ee047328b7d98dc3fb02396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:24.972976 24683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a254ad5ee047328b7d98dc3fb02396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:24.973053 24909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a254ad5ee047328b7d98dc3fb02396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:24.979070 25065 tablet_bootstrap.cc:492] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: Bootstrap starting.
I20260812 01:11:24.980198 25068 tablet_bootstrap.cc:492] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: Bootstrap starting.
I20260812 01:11:24.980504 25071 tablet_bootstrap.cc:492] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41: Bootstrap starting.
I20260812 01:11:24.980964 25065 tablet_bootstrap.cc:654] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:24.982491 25068 tablet_bootstrap.cc:654] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:24.983040 25071 tablet_bootstrap.cc:654] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:24.983743 25065 tablet_bootstrap.cc:492] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: No bootstrap required, opened a new log
I20260812 01:11:24.983940 25065 ts_tablet_manager.cc:1397] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 01:11:24.984990 25065 raft_consensus.cc:348] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:24.985342 25071 tablet_bootstrap.cc:492] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41: No bootstrap required, opened a new log
I20260812 01:11:24.985317 25065 raft_consensus.cc:374] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:24.985431 25065 raft_consensus.cc:729] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ce7bb678944d939eb6f722b518f605, State: Initialized, Role: FOLLOWER
I20260812 01:11:24.985494 25071 ts_tablet_manager.cc:1397] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 01:11:24.985558 25068 tablet_bootstrap.cc:492] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: No bootstrap required, opened a new log
I20260812 01:11:24.985703 25068 ts_tablet_manager.cc:1397] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 01:11:24.985816 25065 consensus_queue.cc:260] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:24.986258 25071 raft_consensus.cc:348] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:24.986514 25071 raft_consensus.cc:374] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:24.986433 25068 raft_consensus.cc:348] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:24.986600 25071 raft_consensus.cc:729] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89c70a93e614d6c9780683b8dc5be41, State: Initialized, Role: FOLLOWER
I20260812 01:11:24.986670 25068 raft_consensus.cc:374] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:24.986370 25076 tablet_replica.cc:439] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } }
I20260812 01:11:24.986753 25068 raft_consensus.cc:729] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7cf253ae04d4789dcf4a626c9a175, State: Initialized, Role: FOLLOWER
I20260812 01:11:24.986796 25076 tablet_replica.cc:442] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:24.986836 25071 consensus_queue.cc:260] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:24.986963 25068 consensus_queue.cc:260] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:24.987430 25071 ts_tablet_manager.cc:1428] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 01:11:24.987588 25068 ts_tablet_manager.cc:1428] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 01:11:24.987438 25077 tablet_replica.cc:439] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } }
I20260812 01:11:24.987771 25077 tablet_replica.cc:442] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:24.987808 25083 tablet_replica.cc:439] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } }
I20260812 01:11:24.988155 25083 tablet_replica.cc:442] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:24.989239 25065 ts_tablet_manager.cc:1428] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
W20260812 01:11:25.022791 24558 tablet.cc:2369] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:25.050385 25077 raft_consensus.cc:482] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:11:25.050643 25077 raft_consensus.cc:504] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.051331 25077 leader_election.cc:288] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977), c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3:43959)
I20260812 01:11:25.051901 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a254ad5ee047328b7d98dc3fb02396" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ce7bb678944d939eb6f722b518f605" is_pre_election: true
I20260812 01:11:25.051930 24942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a254ad5ee047328b7d98dc3fb02396" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89c70a93e614d6c9780683b8dc5be41" is_pre_election: true
I20260812 01:11:25.052145 24463 raft_consensus.cc:2454] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 0.
I20260812 01:11:25.052176 24942 raft_consensus.cc:2454] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 0.
I20260812 01:11:25.052637 24592 leader_election.cc:302] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 77ce7bb678944d939eb6f722b518f605, c89c70a93e614d6c9780683b8dc5be41, dfd7cf253ae04d4789dcf4a626c9a175; no voters:
I20260812 01:11:25.052996 25077 raft_consensus.cc:2781] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:11:25.053113 25077 raft_consensus.cc:482] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:11:25.053198 25077 raft_consensus.cc:3037] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:25.055308 25077 raft_consensus.cc:504] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.056062 25077 leader_election.cc:288] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 election: Requested vote from peers 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977), c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3:43959)
I20260812 01:11:25.056440 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a254ad5ee047328b7d98dc3fb02396" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ce7bb678944d939eb6f722b518f605"
I20260812 01:11:25.056689 24463 raft_consensus.cc:3037] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:25.056630 24942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a254ad5ee047328b7d98dc3fb02396" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89c70a93e614d6c9780683b8dc5be41"
I20260812 01:11:25.056838 24942 raft_consensus.cc:3037] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:25.058734 24463 raft_consensus.cc:2454] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 1.
I20260812 01:11:25.058802 24942 raft_consensus.cc:2454] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 1.
I20260812 01:11:25.059170 24592 leader_election.cc:302] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ce7bb678944d939eb6f722b518f605, dfd7cf253ae04d4789dcf4a626c9a175; no voters:
I20260812 01:11:25.059464 25077 raft_consensus.cc:2781] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:25.059655 25077 raft_consensus.cc:686] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 LEADER]: Becoming Leader. State: Replica: dfd7cf253ae04d4789dcf4a626c9a175, State: Running, Role: LEADER
I20260812 01:11:25.059887 25077 consensus_queue.cc:237] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.061550 25087 tablet_replica.cc:439] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: TxnStatusTablet state changed. Reason: New leader dfd7cf253ae04d4789dcf4a626c9a175. Latest consensus state: current_term: 1 leader_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } }
I20260812 01:11:25.061986 25087 tablet_replica.cc:442] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: This TxnStatusTablet replica's current role is: LEADER
I20260812 01:11:25.062651 25107 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 01:11:25.062908 25107 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 01:11:25.062847 24191 catalog_manager.cc:5495] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2). New cstate: current_term: 1 leader_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } health_report { overall_health: UNKNOWN } } }
I20260812 01:11:25.090780 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.085213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:43355 (local address 127.0.0.1:59820)
0812 01:11:25.085454 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:25.085474 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:25.085620 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:25.086154 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:25.086158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:25.086169 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 01:11:25.086352 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.086361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.087422 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.087428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.088418 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.088431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.089160 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.089167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.089353 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.089418 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 01:11:25.090233 (+ 815us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 01:11:25.090256 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 01:11:25.090335 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 01:11:25.090483 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 01:11:25.090764 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.085182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.20.151.2:43355)
0812 01:11:25.085396 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:25.085402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:25.085550 (+ 148us) server_negotiation.cc:407] Connection header received
0812 01:11:25.085874 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:25.085878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:25.085985 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:25.086130 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:25.086480 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.087259 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.088558 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.088991 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.089118 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.089610 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:25.090122 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:25.090246 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:25.090370 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 01:11:25.090454 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 01:11:25.098572 24463 raft_consensus.cc:1261] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Refusing update from remote peer dfd7cf253ae04d4789dcf4a626c9a175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:25.098629 24942 raft_consensus.cc:1261] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Refusing update from remote peer dfd7cf253ae04d4789dcf4a626c9a175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:25.099581 25087 consensus_queue.cc:1035] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:25.099918 25077 consensus_queue.cc:1035] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:25.109931 25076 tablet_replica.cc:439] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: TxnStatusTablet state changed. Reason: New leader dfd7cf253ae04d4789dcf4a626c9a175. Latest consensus state: current_term: 1 leader_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } }
I20260812 01:11:25.110201 25076 tablet_replica.cc:442] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:25.113442 25077 tablet_replica.cc:439] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } }
I20260812 01:11:25.113749 25077 tablet_replica.cc:442] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: This TxnStatusTablet replica's current role is: LEADER
I20260812 01:11:25.122891 25076 tablet_replica.cc:439] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } }
I20260812 01:11:25.123149 25076 tablet_replica.cc:442] T 45a254ad5ee047328b7d98dc3fb02396 P 77ce7bb678944d939eb6f722b518f605: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:25.131307 25083 tablet_replica.cc:439] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41: TxnStatusTablet state changed. Reason: New leader dfd7cf253ae04d4789dcf4a626c9a175. Latest consensus state: current_term: 1 leader_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } }
I20260812 01:11:25.131608 25083 tablet_replica.cc:442] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:25.132334 25087 tablet_replica.cc:439] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } }
I20260812 01:11:25.132616 25087 tablet_replica.cc:442] T 45a254ad5ee047328b7d98dc3fb02396 P dfd7cf253ae04d4789dcf4a626c9a175: This TxnStatusTablet replica's current role is: LEADER
I20260812 01:11:25.136385 25113 mvcc.cc:205] Tried to move back new op lower bound from 7317492060553519104 to 7317492060407472128. Current Snapshot: MvccSnapshot[applied={T|T < 7317492060553519104}]
I20260812 01:11:25.139219 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.131981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.20.151.2:43355)
0812 01:11:25.132151 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:25.132156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:25.132177 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:11:25.132317 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:25.132324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:25.132410 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:25.132499 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:25.133105 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.134261 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.136389 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.137112 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.137239 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.137932 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:25.138111 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:25.138192 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:25.138754 (+ 562us) server_negotiation.cc:299] Negotiation successful
0812 01:11:25.138847 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 01:11:25.143361 25083 tablet_replica.cc:439] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } } }
I20260812 01:11:25.143632 25083 tablet_replica.cc:442] T 45a254ad5ee047328b7d98dc3fb02396 P c89c70a93e614d6c9780683b8dc5be41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:11:25.158493 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.151621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:58227 (local address 127.20.151.2:43355)
0812 01:11:25.151814 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:25.151821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:25.152563 (+ 742us) server_negotiation.cc:407] Connection header received
0812 01:11:25.152798 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:25.152801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:25.152878 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:25.152950 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:25.153580 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.154328 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.156619 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.157138 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.157235 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.157807 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:25.157924 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:25.157985 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:25.158152 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 01:11:25.158237 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
I20260812 01:11:25.158929 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.151687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:43355 (local address 127.20.151.2:58227)
0812 01:11:25.152455 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:25.152475 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:25.152653 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:25.153020 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:25.153025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:25.153039 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 01:11:25.153433 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.153445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.154450 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.154455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.156485 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.156501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.157318 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.157325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.157517 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.157604 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 01:11:25.158013 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 01:11:25.158025 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 01:11:25.158124 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 01:11:25.158323 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":399,"threads_started":1}
I20260812 01:11:25.169988 24463 raft_consensus.cc:1261] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Refusing update from remote peer dfd7cf253ae04d4789dcf4a626c9a175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:25.170403 24942 raft_consensus.cc:1261] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Refusing update from remote peer dfd7cf253ae04d4789dcf4a626c9a175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:25.170498 25077 consensus_queue.cc:1035] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:25.171027 25127 consensus_queue.cc:1035] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:25.243518 24189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:25.243813 24189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:25.247251 24189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4: Sending DeleteTablet for 3 replicas of tablet 7d0905121a8d446c995b70077cb6220a
I20260812 01:11:25.248555 24909 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d0905121a8d446c995b70077cb6220a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:25 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:54998
I20260812 01:11:25.249053 24426 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d0905121a8d446c995b70077cb6220a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:25 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:34492
I20260812 01:11:25.249467 24682 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d0905121a8d446c995b70077cb6220a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:25 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:59812
I20260812 01:11:25.250033 24189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:11:25.250700 24189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:11:25.258687 25138 tablet_replica.cc:330] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: stopping tablet replica
I20260812 01:11:25.259137 24424 tablet_service.cc:1467] Processing CreateTablet for tablet 2684a771080a4f8484eeb8060dc35979 (DEFAULT_TABLE table=test-table [id=a26e9f2284774a93be3b8009751c67c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:25.259253 25138 raft_consensus.cc:2229] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:11:25.259796 24424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2684a771080a4f8484eeb8060dc35979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:25.260040 25138 raft_consensus.cc:2258] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:25.264029 24682 tablet_service.cc:1467] Processing CreateTablet for tablet 2684a771080a4f8484eeb8060dc35979 (DEFAULT_TABLE table=test-table [id=a26e9f2284774a93be3b8009751c67c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:25.264665 24682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2684a771080a4f8484eeb8060dc35979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:25.264667 24907 tablet_service.cc:1467] Processing CreateTablet for tablet 2684a771080a4f8484eeb8060dc35979 (DEFAULT_TABLE table=test-table [id=a26e9f2284774a93be3b8009751c67c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:25.265132 24907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2684a771080a4f8484eeb8060dc35979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:25.270488 25065 tablet_bootstrap.cc:492] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605: Bootstrap starting.
I20260812 01:11:25.272253 25071 tablet_bootstrap.cc:492] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41: Bootstrap starting.
I20260812 01:11:25.272331 25065 tablet_bootstrap.cc:654] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:25.272603 25137 tablet_replica.cc:330] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605: stopping tablet replica
I20260812 01:11:25.273130 25137 raft_consensus.cc:2229] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:25.273514 25137 raft_consensus.cc:2258] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:25.273957 25071 tablet_bootstrap.cc:654] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:25.274629 25138 ts_tablet_manager.cc:1905] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:25.281080 25137 ts_tablet_manager.cc:1905] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:25.283084 25068 tablet_bootstrap.cc:492] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175: Bootstrap starting.
I20260812 01:11:25.284602 25136 tablet_replica.cc:330] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: stopping tablet replica
I20260812 01:11:25.284873 25068 tablet_bootstrap.cc:654] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:25.285050 25136 raft_consensus.cc:2229] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:25.285442 25136 raft_consensus.cc:2258] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:25.286736 25065 tablet_bootstrap.cc:492] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605: No bootstrap required, opened a new log
I20260812 01:11:25.287127 25065 ts_tablet_manager.cc:1397] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.002s
I20260812 01:11:25.287617 25068 tablet_bootstrap.cc:492] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175: No bootstrap required, opened a new log
I20260812 01:11:25.287811 25068 ts_tablet_manager.cc:1397] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 01:11:25.288360 25065 raft_consensus.cc:348] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.288578 25065 raft_consensus.cc:374] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:25.288658 25065 raft_consensus.cc:729] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ce7bb678944d939eb6f722b518f605, State: Initialized, Role: FOLLOWER
I20260812 01:11:25.288592 25068 raft_consensus.cc:348] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.288843 25068 raft_consensus.cc:374] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:25.288928 25068 raft_consensus.cc:729] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7cf253ae04d4789dcf4a626c9a175, State: Initialized, Role: FOLLOWER
I20260812 01:11:25.288915 25065 consensus_queue.cc:260] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.289201 25068 consensus_queue.cc:260] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.289525 25065 ts_tablet_manager.cc:1428] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 01:11:25.290769 25068 ts_tablet_manager.cc:1428] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 01:11:25.293084 25136 ts_tablet_manager.cc:1905] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:25.296259 25138 ts_tablet_manager.cc:1918] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:11:25.296452 25138 log.cc:1196] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/7d0905121a8d446c995b70077cb6220a
I20260812 01:11:25.297011 25138 ts_tablet_manager.cc:1939] T 7d0905121a8d446c995b70077cb6220a P dfd7cf253ae04d4789dcf4a626c9a175: Deleting consensus metadata
I20260812 01:11:25.298820 24173 catalog_manager.cc:4841] TS dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2:43355): tablet 7d0905121a8d446c995b70077cb6220a (table test-table [id=da64ce5179c94a07a178da862de78297]) successfully deleted
I20260812 01:11:25.299510 25137 ts_tablet_manager.cc:1918] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:11:25.299701 25137 log.cc:1196] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/7d0905121a8d446c995b70077cb6220a
I20260812 01:11:25.300032 25136 ts_tablet_manager.cc:1918] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:11:25.300200 25136 log.cc:1196] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/7d0905121a8d446c995b70077cb6220a
I20260812 01:11:25.300227 25137 ts_tablet_manager.cc:1939] T 7d0905121a8d446c995b70077cb6220a P 77ce7bb678944d939eb6f722b518f605: Deleting consensus metadata
I20260812 01:11:25.300638 25136 ts_tablet_manager.cc:1939] T 7d0905121a8d446c995b70077cb6220a P c89c70a93e614d6c9780683b8dc5be41: Deleting consensus metadata
I20260812 01:11:25.302040 24174 catalog_manager.cc:4841] TS c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3:43959): tablet 7d0905121a8d446c995b70077cb6220a (table test-table [id=da64ce5179c94a07a178da862de78297]) successfully deleted
I20260812 01:11:25.302495 24172 catalog_manager.cc:4841] TS 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977): tablet 7d0905121a8d446c995b70077cb6220a (table test-table [id=da64ce5179c94a07a178da862de78297]) successfully deleted
I20260812 01:11:25.302593 25071 tablet_bootstrap.cc:492] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41: No bootstrap required, opened a new log
I20260812 01:11:25.302762 25071 ts_tablet_manager.cc:1397] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.003s
I20260812 01:11:25.303577 25071 raft_consensus.cc:348] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.303835 25071 raft_consensus.cc:374] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:25.303915 25071 raft_consensus.cc:729] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89c70a93e614d6c9780683b8dc5be41, State: Initialized, Role: FOLLOWER
I20260812 01:11:25.304168 25071 consensus_queue.cc:260] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.304739 25071 ts_tablet_manager.cc:1428] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 01:11:25.411626 25083 raft_consensus.cc:482] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:11:25.411859 25083 raft_consensus.cc:504] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.413440 25083 leader_election.cc:288] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977), dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2:43355)
I20260812 01:11:25.428608 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.413743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41977 (local address 127.20.151.3:48227)
0812 01:11:25.414415 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:25.414436 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:25.414576 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:25.422660 (+ 8084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:25.422666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:25.422700 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:25.423097 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.423113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.424700 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.424706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.426343 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.426356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.427690 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.427696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.427869 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.427907 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 01:11:25.428005 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 01:11:25.428158 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":5,"thread_start_us":642,"threads_started":1}
I20260812 01:11:25.429469 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.414246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:48227 (local address 127.20.151.1:41977)
0812 01:11:25.422294 (+ 8048us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:25.422298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:25.422319 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:11:25.422386 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:25.422390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:25.422454 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:25.422558 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:25.423323 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.424518 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.426511 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.427491 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.427655 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.427816 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:25.428886 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0812 01:11:25.429088 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7933,"thread_start_us":7800,"threads_started":1}
I20260812 01:11:25.430079 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.413865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:34911 (local address 127.20.151.2:43355)
0812 01:11:25.414071 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:25.414077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:25.423390 (+ 9313us) server_negotiation.cc:407] Connection header received
0812 01:11:25.423583 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:25.423591 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:25.423660 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:25.423746 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:25.424266 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.425239 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.427648 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.428938 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.429108 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.429204 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:25.429671 (+ 467us) server_negotiation.cc:299] Negotiation successful
0812 01:11:25.429807 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 01:11:25.430356 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2684a771080a4f8484eeb8060dc35979" candidate_uuid: "c89c70a93e614d6c9780683b8dc5be41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ce7bb678944d939eb6f722b518f605" is_pre_election: true
I20260812 01:11:25.430642 24463 raft_consensus.cc:2454] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c89c70a93e614d6c9780683b8dc5be41 in term 0.
I20260812 01:11:25.430717 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.413876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:43355 (local address 127.20.151.3:34911)
0812 01:11:25.423256 (+ 9380us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:25.423302 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:25.423420 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:25.423796 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:25.423800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:25.423826 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:25.424097 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.424105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.425405 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.425411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.427454 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.427470 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.429357 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.429362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.429490 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.429513 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 01:11:25.430326 (+ 813us) client_negotiation.cc:240] Negotiation successful
0812 01:11:25.430441 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9245,"thread_start_us":8981,"threads_started":1}
I20260812 01:11:25.431207 24826 leader_election.cc:302] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ce7bb678944d939eb6f722b518f605, c89c70a93e614d6c9780683b8dc5be41; no voters:
I20260812 01:11:25.431748 25083 raft_consensus.cc:2781] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:11:25.431897 25083 raft_consensus.cc:482] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:11:25.431751 24719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2684a771080a4f8484eeb8060dc35979" candidate_uuid: "c89c70a93e614d6c9780683b8dc5be41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" is_pre_election: true
I20260812 01:11:25.431999 25083 raft_consensus.cc:3037] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:25.432219 24719 raft_consensus.cc:2454] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c89c70a93e614d6c9780683b8dc5be41 in term 0.
I20260812 01:11:25.435320 25083 raft_consensus.cc:504] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.436074 25083 leader_election.cc:288] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [CANDIDATE]: Term 1 election: Requested vote from peers 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977), dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2:43355)
I20260812 01:11:25.436594 24719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2684a771080a4f8484eeb8060dc35979" candidate_uuid: "c89c70a93e614d6c9780683b8dc5be41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfd7cf253ae04d4789dcf4a626c9a175"
I20260812 01:11:25.436657 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2684a771080a4f8484eeb8060dc35979" candidate_uuid: "c89c70a93e614d6c9780683b8dc5be41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ce7bb678944d939eb6f722b518f605"
I20260812 01:11:25.436797 24719 raft_consensus.cc:3037] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:25.436898 24463 raft_consensus.cc:3037] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:25.439182 24719 raft_consensus.cc:2454] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c89c70a93e614d6c9780683b8dc5be41 in term 1.
I20260812 01:11:25.439704 24827 leader_election.cc:302] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c89c70a93e614d6c9780683b8dc5be41, dfd7cf253ae04d4789dcf4a626c9a175; no voters:
I20260812 01:11:25.439990 24463 raft_consensus.cc:2454] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c89c70a93e614d6c9780683b8dc5be41 in term 1.
I20260812 01:11:25.440044 25083 raft_consensus.cc:2781] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:25.443064 25083 raft_consensus.cc:686] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 1 LEADER]: Becoming Leader. State: Replica: c89c70a93e614d6c9780683b8dc5be41, State: Running, Role: LEADER
I20260812 01:11:25.443547 25083 consensus_queue.cc:237] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:25.447808 24189 catalog_manager.cc:5495] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 reported cstate change: term changed from 0 to 1, leader changed from <none> to c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "c89c70a93e614d6c9780683b8dc5be41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } health_report { overall_health: HEALTHY } } }
I20260812 01:11:25.491400 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.485566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.20.151.3:43959)
0812 01:11:25.486197 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:25.486201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:25.486220 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:11:25.486322 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:25.486326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:25.486408 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:25.486575 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:25.487025 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.487812 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.489103 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.489537 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.489645 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.490098 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:25.490704 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:25.490839 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:25.490933 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 01:11:25.491020 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":353,"threads_started":1}
I20260812 01:11:25.508075 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.502508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:48309 (local address 127.20.151.2:43355)
0812 01:11:25.502689 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:25.502694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:25.503265 (+ 571us) server_negotiation.cc:407] Connection header received
0812 01:11:25.503496 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:25.503500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:25.503571 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:25.503652 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:25.504164 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.504916 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.506252 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.506713 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.506826 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.507316 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:25.507460 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:25.507539 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:25.507670 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 01:11:25.507755 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 01:11:25.508163 25160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.502488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:43355 (local address 127.20.151.3:48309)
0812 01:11:25.503152 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:25.503168 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:25.503348 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:25.503675 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:25.503680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:25.503691 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 01:11:25.504014 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.504025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.505040 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.505044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.506112 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.506124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.506891 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.506897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.507045 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.507103 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 01:11:25.507557 (+ 454us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 01:11:25.507570 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 01:11:25.507639 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 01:11:25.507790 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":340,"threads_started":1}
I20260812 01:11:25.519520 24719 raft_consensus.cc:1261] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Refusing update from remote peer c89c70a93e614d6c9780683b8dc5be41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:25.519721 24463 raft_consensus.cc:1261] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Refusing update from remote peer c89c70a93e614d6c9780683b8dc5be41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:25.520438 25148 consensus_queue.cc:1035] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:25.520906 25083 consensus_queue.cc:1035] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:25.570336 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.564306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:43959 (local address 127.20.151.2:58089)
0812 01:11:25.564566 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:25.564587 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:25.564716 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:25.565055 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:25.565060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:25.565082 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 01:11:25.565301 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.565313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.566344 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.566350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.568037 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:25.568054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.568985 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.568992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:25.569229 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.569291 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 01:11:25.569721 (+ 430us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 01:11:25.569735 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 01:11:25.569823 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 01:11:25.569991 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 01:11:25.571009 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:25.564378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:58089 (local address 127.20.151.3:43959)
0812 01:11:25.564536 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:25.564542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:25.564653 (+ 111us) server_negotiation.cc:407] Connection header received
0812 01:11:25.564850 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:25.564856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:25.564943 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:25.565039 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:25.565439 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.566199 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.568208 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:25.568806 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:25.568946 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:25.569459 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:25.569605 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:25.569698 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:25.570629 (+ 931us) server_negotiation.cc:299] Negotiation successful
0812 01:11:25.570719 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 01:11:25.572212 25113 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 01:11:27.174888 24189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:27.175165 24189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:27.180464 24189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4: Sending DeleteTablet for 3 replicas of tablet 2684a771080a4f8484eeb8060dc35979
I20260812 01:11:27.181496 24426 tablet_service.cc:1514] Processing DeleteTablet for tablet 2684a771080a4f8484eeb8060dc35979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:27 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:34492
I20260812 01:11:27.181524 24909 tablet_service.cc:1514] Processing DeleteTablet for tablet 2684a771080a4f8484eeb8060dc35979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:27 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:54998
I20260812 01:11:27.181587 24682 tablet_service.cc:1514] Processing DeleteTablet for tablet 2684a771080a4f8484eeb8060dc35979 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:27 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:59812
I20260812 01:11:27.194003 25210 tablet_replica.cc:330] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605: stopping tablet replica
I20260812 01:11:27.194443 25211 tablet_replica.cc:330] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41: stopping tablet replica
I20260812 01:11:27.194502 25210 raft_consensus.cc:2229] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:27.194793 25211 raft_consensus.cc:2229] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:11:27.194928 25210 raft_consensus.cc:2258] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:27.195665 25211 raft_consensus.cc:2258] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:27.196352 25210 ts_tablet_manager.cc:1905] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:27.196764 25211 ts_tablet_manager.cc:1905] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:27.197332 25213 tablet_replica.cc:330] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175: stopping tablet replica
I20260812 01:11:27.197777 25213 raft_consensus.cc:2229] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:27.198191 25213 raft_consensus.cc:2258] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:27.199318 25213 ts_tablet_manager.cc:1905] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:27.203929 25210 ts_tablet_manager.cc:1918] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 01:11:27.204263 25210 log.cc:1196] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/2684a771080a4f8484eeb8060dc35979
I20260812 01:11:27.204306 25211 ts_tablet_manager.cc:1918] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 01:11:27.204531 25211 log.cc:1196] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/2684a771080a4f8484eeb8060dc35979
I20260812 01:11:27.205001 25210 ts_tablet_manager.cc:1939] T 2684a771080a4f8484eeb8060dc35979 P 77ce7bb678944d939eb6f722b518f605: Deleting consensus metadata
I20260812 01:11:27.205084 25211 ts_tablet_manager.cc:1939] T 2684a771080a4f8484eeb8060dc35979 P c89c70a93e614d6c9780683b8dc5be41: Deleting consensus metadata
I20260812 01:11:27.206002 25213 ts_tablet_manager.cc:1918] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 01:11:27.206256 25213 log.cc:1196] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/2684a771080a4f8484eeb8060dc35979
I20260812 01:11:27.206346 24174 catalog_manager.cc:4841] TS c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3:43959): tablet 2684a771080a4f8484eeb8060dc35979 (table test-table [id=a26e9f2284774a93be3b8009751c67c0]) successfully deleted
I20260812 01:11:27.206346 24172 catalog_manager.cc:4841] TS 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977): tablet 2684a771080a4f8484eeb8060dc35979 (table test-table [id=a26e9f2284774a93be3b8009751c67c0]) successfully deleted
I20260812 01:11:27.206820 25213 ts_tablet_manager.cc:1939] T 2684a771080a4f8484eeb8060dc35979 P dfd7cf253ae04d4789dcf4a626c9a175: Deleting consensus metadata
I20260812 01:11:27.208130 24173 catalog_manager.cc:4841] TS dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2:43355): tablet 2684a771080a4f8484eeb8060dc35979 (table test-table [id=a26e9f2284774a93be3b8009751c67c0]) successfully deleted
Aug 12 01:11:27 dist-test-slave-21b9 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 01:11:27 dist-test-slave-21b9 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:27.252158 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:27.236857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.20.151.62:46757)
0812 01:11:27.237468 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:27.237472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:27.237497 (+ 25us) server_negotiation.cc:407] Connection header received
0812 01:11:27.237639 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:27.237642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:27.237711 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:27.237814 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:27.239084 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:27.239938 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:27.241254 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:27.241734 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:27.241840 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:27.246435 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:27.246442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:27.246445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:27.246484 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:27.249488 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:27.250291 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:27.250294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:27.250296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:27.250464 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:27.250761 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:27.250764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:27.250767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:27.251069 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:27.251359 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:27.251632 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 01:11:27.251873 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":278,"threads_started":1}
I20260812 01:11:27.255594 24189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:11:27.256242 24189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:11:27.263123 24426 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7a87d583254b96b230cb680d99e31c (DEFAULT_TABLE table=test-table [id=93267f24a6a844ffbefcaf02c63823b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:27.263078 24682 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7a87d583254b96b230cb680d99e31c (DEFAULT_TABLE table=test-table [id=93267f24a6a844ffbefcaf02c63823b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:27.263577 24426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7a87d583254b96b230cb680d99e31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:27.263659 24682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7a87d583254b96b230cb680d99e31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:27.269567 24909 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7a87d583254b96b230cb680d99e31c (DEFAULT_TABLE table=test-table [id=93267f24a6a844ffbefcaf02c63823b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:27.270099 24909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7a87d583254b96b230cb680d99e31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:27.270584 25233 tablet_bootstrap.cc:492] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175: Bootstrap starting.
I20260812 01:11:27.272830 25233 tablet_bootstrap.cc:654] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:27.276153 25232 tablet_bootstrap.cc:492] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605: Bootstrap starting.
I20260812 01:11:27.278867 25235 tablet_bootstrap.cc:492] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41: Bootstrap starting.
I20260812 01:11:27.278937 25232 tablet_bootstrap.cc:654] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:27.280745 25233 tablet_bootstrap.cc:492] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175: No bootstrap required, opened a new log
I20260812 01:11:27.280902 25233 ts_tablet_manager.cc:1397] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260812 01:11:27.281282 25235 tablet_bootstrap.cc:654] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:27.281821 25232 tablet_bootstrap.cc:492] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605: No bootstrap required, opened a new log
I20260812 01:11:27.282064 25232 ts_tablet_manager.cc:1397] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260812 01:11:27.281949 25233 raft_consensus.cc:348] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.282248 25233 raft_consensus.cc:374] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:27.282384 25233 raft_consensus.cc:729] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7cf253ae04d4789dcf4a626c9a175, State: Initialized, Role: FOLLOWER
I20260812 01:11:27.282683 25233 consensus_queue.cc:260] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.283321 25232 raft_consensus.cc:348] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.283584 25232 raft_consensus.cc:374] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:27.283789 25232 raft_consensus.cc:729] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ce7bb678944d939eb6f722b518f605, State: Initialized, Role: FOLLOWER
I20260812 01:11:27.284117 25233 ts_tablet_manager.cc:1428] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 01:11:27.284161 25232 consensus_queue.cc:260] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.285277 25232 ts_tablet_manager.cc:1428] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 01:11:27.285919 25235 tablet_bootstrap.cc:492] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41: No bootstrap required, opened a new log
I20260812 01:11:27.286084 25235 ts_tablet_manager.cc:1397] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260812 01:11:27.286916 25235 raft_consensus.cc:348] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.287256 25235 raft_consensus.cc:374] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:27.287432 25235 raft_consensus.cc:729] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89c70a93e614d6c9780683b8dc5be41, State: Initialized, Role: FOLLOWER
I20260812 01:11:27.287740 25235 consensus_queue.cc:260] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.288410 25235 ts_tablet_manager.cc:1428] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 01:11:27.430260 25199 raft_consensus.cc:482] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:11:27.430589 25199 raft_consensus.cc:504] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.431385 25199 leader_election.cc:288] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977), c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3:43959)
I20260812 01:11:27.432052 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7a87d583254b96b230cb680d99e31c" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ce7bb678944d939eb6f722b518f605" is_pre_election: true
I20260812 01:11:27.432045 24939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7a87d583254b96b230cb680d99e31c" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89c70a93e614d6c9780683b8dc5be41" is_pre_election: true
I20260812 01:11:27.432284 24463 raft_consensus.cc:2454] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 0.
I20260812 01:11:27.432404 24939 raft_consensus.cc:2454] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 0.
I20260812 01:11:27.432823 24592 leader_election.cc:302] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ce7bb678944d939eb6f722b518f605, dfd7cf253ae04d4789dcf4a626c9a175; no voters:
I20260812 01:11:27.433161 25199 raft_consensus.cc:2781] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:11:27.433348 25199 raft_consensus.cc:482] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:11:27.433503 25199 raft_consensus.cc:3037] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:27.436403 25199 raft_consensus.cc:504] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.437083 25199 leader_election.cc:288] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 election: Requested vote from peers 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977), c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3:43959)
I20260812 01:11:27.437726 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7a87d583254b96b230cb680d99e31c" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ce7bb678944d939eb6f722b518f605"
I20260812 01:11:27.437783 24939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7a87d583254b96b230cb680d99e31c" candidate_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89c70a93e614d6c9780683b8dc5be41"
I20260812 01:11:27.438021 24939 raft_consensus.cc:3037] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:27.438071 24463 raft_consensus.cc:3037] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:27.441321 24463 raft_consensus.cc:2454] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 1.
I20260812 01:11:27.441596 24939 raft_consensus.cc:2454] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd7cf253ae04d4789dcf4a626c9a175 in term 1.
I20260812 01:11:27.441764 24592 leader_election.cc:302] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ce7bb678944d939eb6f722b518f605, dfd7cf253ae04d4789dcf4a626c9a175; no voters:
I20260812 01:11:27.442085 25199 raft_consensus.cc:2781] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:27.442248 25199 raft_consensus.cc:686] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 LEADER]: Becoming Leader. State: Replica: dfd7cf253ae04d4789dcf4a626c9a175, State: Running, Role: LEADER
I20260812 01:11:27.442642 25199 consensus_queue.cc:237] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.445899 24190 catalog_manager.cc:5495] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2). New cstate: current_term: 1 leader_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } health_report { overall_health: UNKNOWN } } }
I20260812 01:11:27.479565 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:27.473372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.20.151.2:43355)
0812 01:11:27.474217 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:27.474221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:27.474242 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:11:27.474298 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:27.474302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:27.474370 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:27.474471 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:27.475112 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:27.476220 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:27.477687 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:27.478137 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:27.478249 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:27.478653 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:27.478814 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:27.478893 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:27.479055 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 01:11:27.479133 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":486,"threads_started":1}
I20260812 01:11:27.489560 24939 raft_consensus.cc:1261] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Refusing update from remote peer dfd7cf253ae04d4789dcf4a626c9a175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:27.489801 24463 raft_consensus.cc:1261] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Refusing update from remote peer dfd7cf253ae04d4789dcf4a626c9a175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:27.490175 25187 consensus_queue.cc:1035] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:27.490541 25199 consensus_queue.cc:1035] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:27.495000 25171 mvcc.cc:205] Tried to move back new op lower bound from 7317492070352236544 to 7317492070167183360. Current Snapshot: MvccSnapshot[applied={T|T < 7317492070352236544}]
I20260812 01:11:27.541483 24191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:27.541858 24191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:27.545321 24191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4: Sending DeleteTablet for 3 replicas of tablet 8a7a87d583254b96b230cb680d99e31c
I20260812 01:11:27.546096 24426 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7a87d583254b96b230cb680d99e31c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:27 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:34492
I20260812 01:11:27.546171 24682 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7a87d583254b96b230cb680d99e31c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:27 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:59812
I20260812 01:11:27.546487 25210 tablet_replica.cc:330] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605: stopping tablet replica
I20260812 01:11:27.546458 24909 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7a87d583254b96b230cb680d99e31c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:27 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:54998
I20260812 01:11:27.546722 25210 raft_consensus.cc:2229] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:27.546775 25211 tablet_replica.cc:330] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41: stopping tablet replica
I20260812 01:11:27.546969 25211 raft_consensus.cc:2229] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:27.547060 25210 raft_consensus.cc:2258] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:27.547145 25211 raft_consensus.cc:2258] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:27.547392 25213 tablet_replica.cc:330] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175: stopping tablet replica
I20260812 01:11:27.547703 25213 raft_consensus.cc:2229] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:11:27.547991 25213 raft_consensus.cc:2258] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:27.548249 25211 ts_tablet_manager.cc:1905] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:27.549204 25213 ts_tablet_manager.cc:1905] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:27.549757 24191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:11:27.550426 24191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:11:27.556068 25211 ts_tablet_manager.cc:1918] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:11:27.556367 25211 log.cc:1196] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/8a7a87d583254b96b230cb680d99e31c
I20260812 01:11:27.556914 25211 ts_tablet_manager.cc:1939] T 8a7a87d583254b96b230cb680d99e31c P c89c70a93e614d6c9780683b8dc5be41: Deleting consensus metadata
I20260812 01:11:27.557317 25210 ts_tablet_manager.cc:1905] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:27.558432 24426 tablet_service.cc:1467] Processing CreateTablet for tablet 62a57376287b49ac9af5e5dfc570bf53 (DEFAULT_TABLE table=test-table [id=7e56608b6e304b5b96d3713c20f70a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:27.558668 24682 tablet_service.cc:1467] Processing CreateTablet for tablet 62a57376287b49ac9af5e5dfc570bf53 (DEFAULT_TABLE table=test-table [id=7e56608b6e304b5b96d3713c20f70a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:27.558852 24426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a57376287b49ac9af5e5dfc570bf53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:27.559408 24174 catalog_manager.cc:4841] TS c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3:43959): tablet 8a7a87d583254b96b230cb680d99e31c (table test-table [id=93267f24a6a844ffbefcaf02c63823b0]) successfully deleted
I20260812 01:11:27.559789 24909 tablet_service.cc:1467] Processing CreateTablet for tablet 62a57376287b49ac9af5e5dfc570bf53 (DEFAULT_TABLE table=test-table [id=7e56608b6e304b5b96d3713c20f70a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:27.560348 24909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a57376287b49ac9af5e5dfc570bf53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:27.565914 25235 tablet_bootstrap.cc:492] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41: Bootstrap starting.
I20260812 01:11:27.566913 25210 ts_tablet_manager.cc:1918] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:11:27.567147 25210 log.cc:1196] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/8a7a87d583254b96b230cb680d99e31c
I20260812 01:11:27.567963 24682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a57376287b49ac9af5e5dfc570bf53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:27.568204 25235 tablet_bootstrap.cc:654] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:27.568307 25210 ts_tablet_manager.cc:1939] T 8a7a87d583254b96b230cb680d99e31c P 77ce7bb678944d939eb6f722b518f605: Deleting consensus metadata
I20260812 01:11:27.568835 25232 tablet_bootstrap.cc:492] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605: Bootstrap starting.
I20260812 01:11:27.569456 24172 catalog_manager.cc:4841] TS 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977): tablet 8a7a87d583254b96b230cb680d99e31c (table test-table [id=93267f24a6a844ffbefcaf02c63823b0]) successfully deleted
I20260812 01:11:27.567971 25213 ts_tablet_manager.cc:1918] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:11:27.569891 25213 log.cc:1196] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/8a7a87d583254b96b230cb680d99e31c
I20260812 01:11:27.570538 25213 ts_tablet_manager.cc:1939] T 8a7a87d583254b96b230cb680d99e31c P dfd7cf253ae04d4789dcf4a626c9a175: Deleting consensus metadata
I20260812 01:11:27.571108 25232 tablet_bootstrap.cc:654] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:27.571894 24173 catalog_manager.cc:4841] TS dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2:43355): tablet 8a7a87d583254b96b230cb680d99e31c (table test-table [id=93267f24a6a844ffbefcaf02c63823b0]) successfully deleted
I20260812 01:11:27.573175 25235 tablet_bootstrap.cc:492] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41: No bootstrap required, opened a new log
I20260812 01:11:27.573345 25235 ts_tablet_manager.cc:1397] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260812 01:11:27.573978 25235 raft_consensus.cc:348] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.574209 25235 raft_consensus.cc:374] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:27.574311 25235 raft_consensus.cc:729] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89c70a93e614d6c9780683b8dc5be41, State: Initialized, Role: FOLLOWER
I20260812 01:11:27.574522 25235 consensus_queue.cc:260] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.574816 25233 tablet_bootstrap.cc:492] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175: Bootstrap starting.
I20260812 01:11:27.575237 25235 ts_tablet_manager.cc:1428] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 01:11:27.576512 25232 tablet_bootstrap.cc:492] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605: No bootstrap required, opened a new log
I20260812 01:11:27.576668 25232 ts_tablet_manager.cc:1397] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260812 01:11:27.576900 25233 tablet_bootstrap.cc:654] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:27.577342 25232 raft_consensus.cc:348] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.577574 25232 raft_consensus.cc:374] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:27.577663 25232 raft_consensus.cc:729] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ce7bb678944d939eb6f722b518f605, State: Initialized, Role: FOLLOWER
I20260812 01:11:27.577966 25232 consensus_queue.cc:260] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.578615 25232 ts_tablet_manager.cc:1428] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 01:11:27.579584 25233 tablet_bootstrap.cc:492] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175: No bootstrap required, opened a new log
I20260812 01:11:27.579835 25233 ts_tablet_manager.cc:1397] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 01:11:27.580773 25233 raft_consensus.cc:348] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.581131 25233 raft_consensus.cc:374] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:27.581295 25233 raft_consensus.cc:729] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7cf253ae04d4789dcf4a626c9a175, State: Initialized, Role: FOLLOWER
I20260812 01:11:27.581723 25233 consensus_queue.cc:260] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.582362 25233 ts_tablet_manager.cc:1428] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 01:11:27.922353 25201 raft_consensus.cc:482] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:11:27.922583 25201 raft_consensus.cc:504] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.923313 25201 leader_election.cc:288] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977), dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2:43355)
I20260812 01:11:27.923955 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a57376287b49ac9af5e5dfc570bf53" candidate_uuid: "c89c70a93e614d6c9780683b8dc5be41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ce7bb678944d939eb6f722b518f605" is_pre_election: true
I20260812 01:11:27.924034 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a57376287b49ac9af5e5dfc570bf53" candidate_uuid: "c89c70a93e614d6c9780683b8dc5be41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" is_pre_election: true
I20260812 01:11:27.924269 24463 raft_consensus.cc:2454] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c89c70a93e614d6c9780683b8dc5be41 in term 0.
I20260812 01:11:27.924434 24718 raft_consensus.cc:2454] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c89c70a93e614d6c9780683b8dc5be41 in term 0.
I20260812 01:11:27.924896 24826 leader_election.cc:302] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ce7bb678944d939eb6f722b518f605, c89c70a93e614d6c9780683b8dc5be41; no voters:
I20260812 01:11:27.925189 25201 raft_consensus.cc:2781] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:11:27.925427 25201 raft_consensus.cc:482] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:11:27.925552 25201 raft_consensus.cc:3037] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:27.928200 25201 raft_consensus.cc:504] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.928732 25201 leader_election.cc:288] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [CANDIDATE]: Term 1 election: Requested vote from peers 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977), dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2:43355)
I20260812 01:11:27.929431 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a57376287b49ac9af5e5dfc570bf53" candidate_uuid: "c89c70a93e614d6c9780683b8dc5be41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ce7bb678944d939eb6f722b518f605"
I20260812 01:11:27.929455 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a57376287b49ac9af5e5dfc570bf53" candidate_uuid: "c89c70a93e614d6c9780683b8dc5be41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfd7cf253ae04d4789dcf4a626c9a175"
I20260812 01:11:27.929759 24463 raft_consensus.cc:3037] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:27.929852 24718 raft_consensus.cc:3037] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:27.933363 24718 raft_consensus.cc:2454] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c89c70a93e614d6c9780683b8dc5be41 in term 1.
I20260812 01:11:27.933429 24463 raft_consensus.cc:2454] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c89c70a93e614d6c9780683b8dc5be41 in term 1.
I20260812 01:11:27.933946 24827 leader_election.cc:302] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c89c70a93e614d6c9780683b8dc5be41, dfd7cf253ae04d4789dcf4a626c9a175; no voters:
I20260812 01:11:27.934262 25201 raft_consensus.cc:2781] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:27.934741 25201 raft_consensus.cc:686] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 1 LEADER]: Becoming Leader. State: Replica: c89c70a93e614d6c9780683b8dc5be41, State: Running, Role: LEADER
I20260812 01:11:27.935072 25201 consensus_queue.cc:237] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } }
I20260812 01:11:27.937974 24189 catalog_manager.cc:5495] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 reported cstate change: term changed from 0 to 1, leader changed from <none> to c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "c89c70a93e614d6c9780683b8dc5be41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c89c70a93e614d6c9780683b8dc5be41" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 43959 } health_report { overall_health: HEALTHY } } }
I20260812 01:11:27.960775 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:27.954417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.20.151.3:43959)
0812 01:11:27.955093 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:27.955097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:27.955117 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:11:27.955169 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:27.955173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:27.955232 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:27.955353 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:27.955941 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:27.957074 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:27.958596 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:27.959118 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:27.959229 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:27.959954 (+ 725us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:27.960086 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:27.960169 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:27.960316 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 01:11:27.960392 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":306,"threads_started":1}
I20260812 01:11:27.970814 24463 raft_consensus.cc:1261] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Refusing update from remote peer c89c70a93e614d6c9780683b8dc5be41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:27.970814 24718 raft_consensus.cc:1261] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Refusing update from remote peer c89c70a93e614d6c9780683b8dc5be41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:27.971349 25201 consensus_queue.cc:1035] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ce7bb678944d939eb6f722b518f605" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:27.971724 25256 consensus_queue.cc:1035] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfd7cf253ae04d4789dcf4a626c9a175" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 01:11:27.993363 25176 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 01:11:29.582028 24190 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:29.582356 24190 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:29.587001 24190 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a58848daa399457897c15a91f6eea8b4: Sending DeleteTablet for 3 replicas of tablet 62a57376287b49ac9af5e5dfc570bf53
I20260812 01:11:29.587941 24426 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a57376287b49ac9af5e5dfc570bf53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:29 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:34492
I20260812 01:11:29.588143 24909 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a57376287b49ac9af5e5dfc570bf53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:29 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:54998
I20260812 01:11:29.588177 24682 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a57376287b49ac9af5e5dfc570bf53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:29 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:59812
I20260812 01:11:29.588799 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24303
I20260812 01:11:29.600637 25306 tablet_replica.cc:330] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605: stopping tablet replica
I20260812 01:11:29.601126 25308 tablet_replica.cc:330] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175: stopping tablet replica
I20260812 01:11:29.601163 25306 raft_consensus.cc:2229] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:29.601558 25308 raft_consensus.cc:2229] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:29.601666 25306 raft_consensus.cc:2258] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:29.602336 25308 raft_consensus.cc:2258] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:29.603335 25306 ts_tablet_manager.cc:1905] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:29.603494 25308 ts_tablet_manager.cc:1905] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:29.603663 25307 tablet_replica.cc:330] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41: stopping tablet replica
I20260812 01:11:29.604058 25307 raft_consensus.cc:2229] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:11:29.604655 25307 raft_consensus.cc:2258] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:29.606149 25307 ts_tablet_manager.cc:1905] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:29.610918 25308 ts_tablet_manager.cc:1918] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 01:11:29.611181 25308 log.cc:1196] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/62a57376287b49ac9af5e5dfc570bf53
I20260812 01:11:29.611811 25308 ts_tablet_manager.cc:1939] T 62a57376287b49ac9af5e5dfc570bf53 P dfd7cf253ae04d4789dcf4a626c9a175: Deleting consensus metadata
I20260812 01:11:29.612025 25306 ts_tablet_manager.cc:1918] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 01:11:29.612259 25306 log.cc:1196] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/62a57376287b49ac9af5e5dfc570bf53
I20260812 01:11:29.612793 25306 ts_tablet_manager.cc:1939] T 62a57376287b49ac9af5e5dfc570bf53 P 77ce7bb678944d939eb6f722b518f605: Deleting consensus metadata
I20260812 01:11:29.613121 24173 catalog_manager.cc:4841] TS dfd7cf253ae04d4789dcf4a626c9a175 (127.20.151.2:43355): tablet 62a57376287b49ac9af5e5dfc570bf53 (table test-table [id=7e56608b6e304b5b96d3713c20f70a3b]) successfully deleted
I20260812 01:11:29.613725 25307 ts_tablet_manager.cc:1918] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 01:11:29.614024 25307 log.cc:1196] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/62a57376287b49ac9af5e5dfc570bf53
I20260812 01:11:29.614017 24172 catalog_manager.cc:4841] TS 77ce7bb678944d939eb6f722b518f605 (127.20.151.1:41977): tablet 62a57376287b49ac9af5e5dfc570bf53 (table test-table [id=7e56608b6e304b5b96d3713c20f70a3b]) successfully deleted
I20260812 01:11:29.614676 25307 ts_tablet_manager.cc:1939] T 62a57376287b49ac9af5e5dfc570bf53 P c89c70a93e614d6c9780683b8dc5be41: Deleting consensus metadata
I20260812 01:11:29.615801 24174 catalog_manager.cc:4841] TS c89c70a93e614d6c9780683b8dc5be41 (127.20.151.3:43959): tablet 62a57376287b49ac9af5e5dfc570bf53 (table test-table [id=7e56608b6e304b5b96d3713c20f70a3b]) successfully deleted
Aug 12 01:11:29 dist-test-slave-21b9 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:29.630012 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:29.590888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.20.151.1:41977)
0812 01:11:29.602441 (+ 11553us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:29.602449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:29.607937 (+ 5488us) server_negotiation.cc:407] Connection header received
0812 01:11:29.608144 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:29.608149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:29.608232 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:29.608337 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:29.610577 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:29.611929 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:29.614061 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:29.614777 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:29.614913 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:29.622977 (+ 8064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:29.622997 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:29.623014 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:29.623057 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:29.626812 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:29.627662 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:29.627670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:29.627673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:29.627867 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:29.628292 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:29.628296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:29.628299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:29.628737 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:29.629039 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:29.629361 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 01:11:29.629608 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11338,"thread_start_us":11237,"threads_started":1}
W20260812 01:11:29.630698 24549 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49600
W20260812 01:11:29.631467 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24303: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:11:29.631691 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 24303
I20260812 01:11:29.663923 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24562
Aug 12 01:11:29 dist-test-slave-21b9 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:11:29.680516 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:29.664836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.20.151.2:43355)
0812 01:11:29.665394 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:29.665399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:29.665415 (+ 16us) server_negotiation.cc:407] Connection header received
0812 01:11:29.665470 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:29.665474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:29.665539 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:29.665630 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:29.666986 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:29.667825 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:29.669110 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:29.669561 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:29.669659 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:29.674081 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:29.674099 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:29.674117 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:29.674155 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:29.677347 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:29.678101 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:29.678108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:29.678111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:29.678276 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:29.678666 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:29.678671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:29.678673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:29.679095 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:29.679407 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:29.679943 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 01:11:29.680148 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":269,"threads_started":1}
W20260812 01:11:29.681275 24782 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59764
W20260812 01:11:29.682042 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24562: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:11:29.682237 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 24562
I20260812 01:11:29.720208 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24793
Aug 12 01:11:29 dist-test-slave-21b9 krb5kdc[24111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:11:29.736658 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:29.721086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.20.151.3:43959)
0812 01:11:29.721620 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:29.721625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:29.721640 (+ 15us) server_negotiation.cc:407] Connection header received
0812 01:11:29.721697 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:29.721701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:29.721761 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:29.721856 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:29.723260 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:29.724064 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:29.725336 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:29.725767 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:29.725880 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:29.730480 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:29.730500 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:29.730519 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:29.730560 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:29.733632 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:29.734389 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:29.734397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:29.734400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:29.734562 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:29.734938 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:29.734943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:29.734945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:29.735449 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:29.735746 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:29.736066 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 01:11:29.736316 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":241,"threads_started":1}
W20260812 01:11:29.737390 25027 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49890
W20260812 01:11:29.738226 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24793: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:11:29.738407 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 24793
I20260812 01:11:29.768492 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24145
I20260812 01:11:29.769797 24257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:29.891101 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 24145
2026-08-12T01:11:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8925 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:11:30 dist-test-slave-21b9 krb5kdc[25322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:11:30 dist-test-slave-21b9 krb5kdc[25322](info): set up 2 sockets
Aug 12 01:11:30 dist-test-slave-21b9 krb5kdc[25322](info): commencing operation
krb5kdc: starting...
W20260812 01:11:31.149627 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:11:31 dist-test-slave-21b9 krb5kdc[25322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:11:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:11:31.395740 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:34089
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:34089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:31.738032 25355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:31.738348 25355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:31.738484 25355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:31.745910 25355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:31.746039 25355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:31.746095 25355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:31.746191 25355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:31.746253 25355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:11:31.756474 25355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:34089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:34089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:31.757886 25355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:31.759230 25355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:31.771158 25363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:31.771198 25364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:31.771241 25366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:31.772358 25355 server_base.cc:1034] running on GCE node
I20260812 01:11:31.773269 25355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:31.774776 25355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:31.776018 25355 hybrid_clock.cc:648] HybridClock initialized: now 1786497091775936 us; error 49 us; skew 500 ppm
I20260812 01:11:31.776377 25355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:31 dist-test-slave-21b9 krb5kdc[25322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497091, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:31.779739 25355 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:11:31.781595 25355 webserver.cc:460] Webserver started at http://127.20.151.62:44493/ using document root <none> and password file <none>
I20260812 01:11:31.782336 25355 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:31.782461 25355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:31.782778 25355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:31.785861 25355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "577753311ee440968d91e60811dc063b"
format_stamp: "Formatted at 2026-08-12 01:11:31 on dist-test-slave-21b9"
server_key: "e8a3edccfae22b94e6fa1415483540e8"
server_key_iv: "51411dd828c03549d22ff73a8888ecdc"
server_key_version: "encryptionkey@0"
I20260812 01:11:31.786628 25355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "577753311ee440968d91e60811dc063b"
format_stamp: "Formatted at 2026-08-12 01:11:31 on dist-test-slave-21b9"
server_key: "e8a3edccfae22b94e6fa1415483540e8"
server_key_iv: "51411dd828c03549d22ff73a8888ecdc"
server_key_version: "encryptionkey@0"
I20260812 01:11:31.792006 25355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:11:31.795667 25377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:31.797053 25355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:11:31.797240 25355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "577753311ee440968d91e60811dc063b"
format_stamp: "Formatted at 2026-08-12 01:11:31 on dist-test-slave-21b9"
server_key: "e8a3edccfae22b94e6fa1415483540e8"
server_key_iv: "51411dd828c03549d22ff73a8888ecdc"
server_key_version: "encryptionkey@0"
I20260812 01:11:31.797434 25355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:31.811398 25355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:31.815923 25355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:31.816220 25355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:31.834499 25355 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:34089
I20260812 01:11:31.834522 25463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:34089 every 8 connection(s)
I20260812 01:11:31.835986 25355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:31.839465 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 25355
I20260812 01:11:31.839704 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:31.840054 25465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:31.840346 21084 external_mini_cluster.cc:1428] Setting key c289c7e6d0c801beccd03e3f621f6ac2
I20260812 01:11:31.854964 25465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b: Bootstrap starting.
Aug 12 01:11:31 dist-test-slave-21b9 krb5kdc[25322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:31.858952 25465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:31.860390 25465 log.cc:824] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:31.864017 25465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b: No bootstrap required, opened a new log
I20260812 01:11:31.865126 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:31.842312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.20.151.62:34089)
0812 01:11:31.843154 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:31.843166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:31.843212 (+ 46us) server_negotiation.cc:407] Connection header received
0812 01:11:31.844886 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:31.844909 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:31.845325 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:31.845667 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:31.847565 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:31.848876 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:31.850692 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:31.851510 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:31.851669 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:31.856326 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:31.856338 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:31.856358 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:31.856420 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:31.860298 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:31.861230 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:31.861237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:31.861243 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:31.861437 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:31.861887 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:31.861892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:31.861895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:31.862606 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:31.862892 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:31.863229 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 01:11:31.863761 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":350,"threads_started":1}
I20260812 01:11:31.872532 25465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577753311ee440968d91e60811dc063b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34089 } }
I20260812 01:11:31.872901 25465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:31.872982 25465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577753311ee440968d91e60811dc063b, State: Initialized, Role: FOLLOWER
I20260812 01:11:31.873469 25465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577753311ee440968d91e60811dc063b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34089 } }
I20260812 01:11:31.873687 25465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:31.873793 25465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:31.873925 25465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:31.876200 25465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577753311ee440968d91e60811dc063b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34089 } }
I20260812 01:11:31.876852 25465 leader_election.cc:302] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 577753311ee440968d91e60811dc063b; no voters:
I20260812 01:11:31.877405 25465 leader_election.cc:288] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:31.877549 25475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:31.878216 25475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [term 1 LEADER]: Becoming Leader. State: Replica: 577753311ee440968d91e60811dc063b, State: Running, Role: LEADER
I20260812 01:11:31.878729 25475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577753311ee440968d91e60811dc063b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34089 } }
I20260812 01:11:31.879522 25465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:31.881343 25478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "577753311ee440968d91e60811dc063b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577753311ee440968d91e60811dc063b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34089 } } }
I20260812 01:11:31.881831 25478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:31.881731 25482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 577753311ee440968d91e60811dc063b. Latest consensus state: current_term: 1 leader_uuid: "577753311ee440968d91e60811dc063b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577753311ee440968d91e60811dc063b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34089 } } }
I20260812 01:11:31.882006 25482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:31.883625 25490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:31.889027 25490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:31.897876 25490 catalog_manager.cc:1347] Generated new cluster ID: 6c053551e6b643618b44b37f5521493d
I20260812 01:11:31.897986 25490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:31.913367 25490 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:31.914742 25490 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:31.923900 25490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 577753311ee440968d91e60811dc063b: Generated new TSK 0
I20260812 01:11:31.925076 25490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:11:31.990486 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34089
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:32.336539 25515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:32.336859 25515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:32.336992 25515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:32.344352 25515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:32.344504 25515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:32.344663 25515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:11:32.354768 25515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:32.356140 25515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:32.357457 25515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:32.371028 25529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:32.370992 25525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:32.371063 25524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:32.371778 25515 server_base.cc:1034] running on GCE node
I20260812 01:11:32.372496 25515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:32.373469 25515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:32.374715 25515 hybrid_clock.cc:648] HybridClock initialized: now 1786497092374674 us; error 32 us; skew 500 ppm
I20260812 01:11:32.375063 25515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:32 dist-test-slave-21b9 krb5kdc[25322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497092, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:32.378232 25515 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:11:32.379827 25515 webserver.cc:460] Webserver started at http://127.20.151.1:45959/ using document root <none> and password file <none>
I20260812 01:11:32.380573 25515 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:32.380697 25515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:32.380998 25515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:32.384116 25515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "52ba9439972e433483553e862844a325"
format_stamp: "Formatted at 2026-08-12 01:11:32 on dist-test-slave-21b9"
server_key: "926482aeafb2d023c4a5b6e1ad637596"
server_key_iv: "daa0a641822b10902e6e90829f8a3489"
server_key_version: "encryptionkey@0"
I20260812 01:11:32.384951 25515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "52ba9439972e433483553e862844a325"
format_stamp: "Formatted at 2026-08-12 01:11:32 on dist-test-slave-21b9"
server_key: "926482aeafb2d023c4a5b6e1ad637596"
server_key_iv: "daa0a641822b10902e6e90829f8a3489"
server_key_version: "encryptionkey@0"
I20260812 01:11:32.390343 25515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 01:11:32.394255 25540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:32.395738 25515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:11:32.395915 25515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "52ba9439972e433483553e862844a325"
format_stamp: "Formatted at 2026-08-12 01:11:32 on dist-test-slave-21b9"
server_key: "926482aeafb2d023c4a5b6e1ad637596"
server_key_iv: "daa0a641822b10902e6e90829f8a3489"
server_key_version: "encryptionkey@0"
I20260812 01:11:32.396128 25515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:32.429131 25515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:32.433333 25515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:32.433610 25515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:32.434618 25515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:32.436066 25515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:32.436160 25515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:32.436301 25515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:32.436357 25515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:32.468833 25515 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:35311
I20260812 01:11:32.468878 25759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:35311 every 8 connection(s)
I20260812 01:11:32.470191 25515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:11:32.475827 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 25515
I20260812 01:11:32.476045 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:11:32.476852 21084 external_mini_cluster.cc:1428] Setting key b84ea8848598fa09ee8f9ccb87495fbc
Aug 12 01:11:32 dist-test-slave-21b9 krb5kdc[25322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497092, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:32.491477 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:32.472736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:57459 (local address 127.20.151.62:34089)
0812 01:11:32.473204 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:32.473209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:32.474053 (+ 844us) server_negotiation.cc:407] Connection header received
0812 01:11:32.475992 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:32.475998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:32.476090 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:32.476199 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:32.478136 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:32.479111 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:32.481384 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:32.481814 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:32.481919 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:32.485866 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:32.485871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:32.485875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:32.485908 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:32.488390 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:32.489111 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:32.489115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:32.489117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:32.489271 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:32.489714 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:32.489717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:32.489720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:32.489986 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:32.490121 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:32.490839 (+ 718us) server_negotiation.cc:299] Negotiation successful
0812 01:11:32.491079 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":227,"threads_started":1}
I20260812 01:11:32.492449 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:32.473182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34089 (local address 127.20.151.1:57459)
0812 01:11:32.473886 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:32.473925 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:32.475718 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:32.476495 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:32.476509 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:32.477328 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:32.477975 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:32.477994 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:32.479331 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:32.479338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:32.481239 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:32.481256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:32.482053 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:32.482061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:32.482342 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:32.483073 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:32.483125 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:32.485642 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:32.488527 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:32.488537 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:32.488555 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:32.488979 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:32.489382 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:32.489387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:32.489391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:32.489596 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:32.490122 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:32.490130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:32.490463 (+ 333us) client_negotiation.cc:769] Sending connection context
0812 01:11:32.490708 (+ 245us) client_negotiation.cc:240] Negotiation successful
0812 01:11:32.491261 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":278,"threads_started":1}
I20260812 01:11:32.494024 25763 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34089
I20260812 01:11:32.494388 25763 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:32.495236 25763 heartbeater.cc:511] Master 127.20.151.62:34089 requested a full tablet report, sending...
I20260812 01:11:32.497489 25413 ts_manager.cc:194] Registered new tserver with Master: 52ba9439972e433483553e862844a325 (127.20.151.1:35311)
I20260812 01:11:32.498926 25413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:57459
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:11:32.530586 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34089
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:32.877864 25775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:32.878188 25775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:32.878288 25775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:32.885933 25775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:32.886065 25775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:32.886185 25775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:11:32.896574 25775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:32.897893 25775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:32.899195 25775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:32.912876 25780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:32.912829 25781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:32.912794 25787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:32.913705 25775 server_base.cc:1034] running on GCE node
I20260812 01:11:32.914338 25775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:32.915390 25775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:32.916656 25775 hybrid_clock.cc:648] HybridClock initialized: now 1786497092916584 us; error 49 us; skew 500 ppm
I20260812 01:11:32.916981 25775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:32 dist-test-slave-21b9 krb5kdc[25322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497092, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:32.920321 25775 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:11:32.921978 25775 webserver.cc:460] Webserver started at http://127.20.151.2:38975/ using document root <none> and password file <none>
I20260812 01:11:32.922716 25775 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:32.922828 25775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:32.923151 25775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:32.926347 25775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "1b5f113b93ae4a98ab52faa6cf9ee720"
format_stamp: "Formatted at 2026-08-12 01:11:32 on dist-test-slave-21b9"
server_key: "fb2c4f33846bff03cad4d18709979eba"
server_key_iv: "d99e7e2e0b4afb5312ad99321e0b7f66"
server_key_version: "encryptionkey@0"
I20260812 01:11:32.927174 25775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "1b5f113b93ae4a98ab52faa6cf9ee720"
format_stamp: "Formatted at 2026-08-12 01:11:32 on dist-test-slave-21b9"
server_key: "fb2c4f33846bff03cad4d18709979eba"
server_key_iv: "d99e7e2e0b4afb5312ad99321e0b7f66"
server_key_version: "encryptionkey@0"
I20260812 01:11:32.932570 25775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 01:11:32.936215 25802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:32.937585 25775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:32.937752 25775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "1b5f113b93ae4a98ab52faa6cf9ee720"
format_stamp: "Formatted at 2026-08-12 01:11:32 on dist-test-slave-21b9"
server_key: "fb2c4f33846bff03cad4d18709979eba"
server_key_iv: "d99e7e2e0b4afb5312ad99321e0b7f66"
server_key_version: "encryptionkey@0"
I20260812 01:11:32.937950 25775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:32.955000 25775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:32.959510 25775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:32.959801 25775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:32.960884 25775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:32.962319 25775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:32.962399 25775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:32.962548 25775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:32.962639 25775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:32.996197 25775 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:44763
I20260812 01:11:32.996248 25997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:44763 every 8 connection(s)
I20260812 01:11:32.997491 25775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:11:33.006244 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 25775
I20260812 01:11:33.006456 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:11:33.006908 21084 external_mini_cluster.cc:1428] Setting key d1066519ae41d529e0fefbad23bdb490
Aug 12 01:11:33 dist-test-slave-21b9 krb5kdc[25322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497092, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:33.019644 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:33.000157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:49295 (local address 127.20.151.62:34089)
0812 01:11:33.000622 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:33.000642 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:33.001772 (+ 1130us) server_negotiation.cc:407] Connection header received
0812 01:11:33.003733 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:33.003738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:33.003827 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:33.003924 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:33.005501 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:33.006455 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:33.008396 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:33.008924 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:33.009023 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:33.013040 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:33.013046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:33.013051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:33.013101 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:33.016219 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:33.017035 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:33.017040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:33.017043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:33.017212 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:33.017692 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:33.017696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:33.017699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:33.018075 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:33.018208 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:33.019014 (+ 806us) server_negotiation.cc:299] Negotiation successful
0812 01:11:33.019250 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20260812 01:11:33.020704 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:33.000787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34089 (local address 127.20.151.2:49295)
0812 01:11:33.001602 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:33.001654 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:33.003449 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:33.004187 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:33.004195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:33.004741 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:33.005322 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:33.005341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:33.006647 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:33.006651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:33.008240 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:33.008257 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:33.009169 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:33.009176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:33.009391 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:33.010122 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:33.010151 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:33.012771 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:33.016377 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:33.016385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:33.016400 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:33.016902 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:33.017323 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:33.017327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:33.017329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:33.017581 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:33.018201 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:33.018208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:33.018603 (+ 395us) client_negotiation.cc:769] Sending connection context
0812 01:11:33.018875 (+ 272us) client_negotiation.cc:240] Negotiation successful
0812 01:11:33.019462 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":316,"threads_started":1}
I20260812 01:11:33.022171 25999 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34089
I20260812 01:11:33.022464 25999 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:33.023190 25999 heartbeater.cc:511] Master 127.20.151.62:34089 requested a full tablet report, sending...
I20260812 01:11:33.024673 25413 ts_manager.cc:194] Registered new tserver with Master: 1b5f113b93ae4a98ab52faa6cf9ee720 (127.20.151.2:44763)
I20260812 01:11:33.025492 25413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:49295
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:11:33.060318 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34089
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:33.402854 26009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:33.403136 26009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:33.403232 26009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:33.410416 26009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:33.410533 26009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:33.410645 26009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:11:33.420302 26009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:33.421522 26009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:33.422813 26009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:33.436040 26017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:33.436009 26021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:33.435981 26029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:33.437151 26009 server_base.cc:1034] running on GCE node
I20260812 01:11:33.437841 26009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:33.438784 26009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:33.440001 26009 hybrid_clock.cc:648] HybridClock initialized: now 1786497093439947 us; error 41 us; skew 500 ppm
I20260812 01:11:33.440366 26009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:33 dist-test-slave-21b9 krb5kdc[25322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497093, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:33.443813 26009 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:11:33.445415 26009 webserver.cc:460] Webserver started at http://127.20.151.3:38753/ using document root <none> and password file <none>
I20260812 01:11:33.446130 26009 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:33.446254 26009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:33.446573 26009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:33.449575 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "bb8288bff35a4d0f9f14584ae1c348f5"
format_stamp: "Formatted at 2026-08-12 01:11:33 on dist-test-slave-21b9"
server_key: "2dec5947043f08db4ab387889063be04"
server_key_iv: "4ade6764080f0d1157987eba4cf38b4e"
server_key_version: "encryptionkey@0"
I20260812 01:11:33.450340 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "bb8288bff35a4d0f9f14584ae1c348f5"
format_stamp: "Formatted at 2026-08-12 01:11:33 on dist-test-slave-21b9"
server_key: "2dec5947043f08db4ab387889063be04"
server_key_iv: "4ade6764080f0d1157987eba4cf38b4e"
server_key_version: "encryptionkey@0"
I20260812 01:11:33.455583 26009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 01:11:33.459034 26037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:33.460433 26009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:11:33.460605 26009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "bb8288bff35a4d0f9f14584ae1c348f5"
format_stamp: "Formatted at 2026-08-12 01:11:33 on dist-test-slave-21b9"
server_key: "2dec5947043f08db4ab387889063be04"
server_key_iv: "4ade6764080f0d1157987eba4cf38b4e"
server_key_version: "encryptionkey@0"
I20260812 01:11:33.460819 26009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:33.474896 26009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:33.479022 26009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:33.479326 26009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:33.480306 26009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:33.481642 26009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:33.481714 26009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:33.481824 26009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:33.481889 26009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:33.502255 25763 heartbeater.cc:503] Master 127.20.151.62:34089 was elected leader, sending a full tablet report...
I20260812 01:11:33.513710 26009 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:35913
I20260812 01:11:33.513757 26233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:35913 every 8 connection(s)
I20260812 01:11:33.515008 26009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:11:33.521978 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 26009
I20260812 01:11:33.522183 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:11:33.522595 21084 external_mini_cluster.cc:1428] Setting key 07c6736d2e1522f16099ada2ba49942e
Aug 12 01:11:33 dist-test-slave-21b9 krb5kdc[25322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497093, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:33.535154 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:33.517596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:48321 (local address 127.20.151.62:34089)
0812 01:11:33.517786 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:33.517790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:33.518966 (+ 1176us) server_negotiation.cc:407] Connection header received
0812 01:11:33.520853 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:33.520859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:33.520952 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:33.521041 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:33.522768 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:33.523806 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:33.525804 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:33.526215 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:33.526309 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:33.529801 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:33.529806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:33.529810 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:33.529842 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:33.532365 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:33.533079 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:33.533082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:33.533085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:33.533205 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:33.533649 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:33.533652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:33.533654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:33.533920 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:33.534034 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:33.534717 (+ 683us) server_negotiation.cc:299] Negotiation successful
0812 01:11:33.534900 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:11:33.536338 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:33.518090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34089 (local address 127.20.151.3:48321)
0812 01:11:33.518803 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:33.518847 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:33.520582 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:33.521285 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:33.521293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:33.521929 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:33.522568 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:33.522590 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:33.523954 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:33.523959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:33.525670 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:33.525683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:33.526398 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:33.526402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:33.526606 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:33.527223 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:33.527246 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:33.529581 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:33.532509 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:33.532517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:33.532532 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:33.532979 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:33.533327 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:33.533331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:33.533333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:33.533559 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:33.534047 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:33.534053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:33.534418 (+ 365us) client_negotiation.cc:769] Sending connection context
0812 01:11:33.534659 (+ 241us) client_negotiation.cc:240] Negotiation successful
0812 01:11:33.535091 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":267,"threads_started":1}
I20260812 01:11:33.537752 26235 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34089
I20260812 01:11:33.538071 26235 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:33.538866 26235 heartbeater.cc:511] Master 127.20.151.62:34089 requested a full tablet report, sending...
I20260812 01:11:33.540367 25412 ts_manager.cc:194] Registered new tserver with Master: bb8288bff35a4d0f9f14584ae1c348f5 (127.20.151.3:35913)
I20260812 01:11:33.541278 25412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:48321
I20260812 01:11:33.551002 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 01:11:33.593395 26003 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:11:33.590705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.20.151.62:34089)
0812 01:11:33.590899 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:33.590903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:33.591298 (+ 395us) server_negotiation.cc:407] Connection header received
0812 01:11:33.591511 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:33.591515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:33.591618 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:33.591727 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:33.593172 (+ 1445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56716: BlockingRecv error: recv got EOF from 127.0.0.1:56716 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:11:33.596513 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25515
W20260812 01:11:33.599457 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25515: Not authorized: Client connection negotiation failed: client connection to 127.20.151.1:35311: server requires authentication, but client does not have Kerberos credentials available
I20260812 01:11:33.599659 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 25515
I20260812 01:11:33.629283 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25775
W20260812 01:11:33.632216 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25775: Not authorized: Client connection negotiation failed: client connection to 127.20.151.2:44763: server requires authentication, but client does not have Kerberos credentials available
I20260812 01:11:33.632406 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 25775
I20260812 01:11:33.659168 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26009
W20260812 01:11:33.662289 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26009: Not authorized: Client connection negotiation failed: client connection to 127.20.151.3:35913: server requires authentication, but client does not have Kerberos credentials available
I20260812 01:11:33.662448 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 26009
I20260812 01:11:33.685593 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25355
I20260812 01:11:33.686867 25458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:33.794574 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 25355
2026-08-12T01:11:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3895 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:11:33 dist-test-slave-21b9 krb5kdc[26273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:11:33 dist-test-slave-21b9 krb5kdc[26273](info): set up 2 sockets
Aug 12 01:11:33 dist-test-slave-21b9 krb5kdc[26273](info): commencing operation
krb5kdc: starting...
W20260812 01:11:35.051133 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:11:35 dist-test-slave-21b9 krb5kdc[26273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:11:35.300555 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:45125
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:45125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:35.642992 26294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:35.643376 26294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:35.643512 26294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:35.650991 26294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:35.651126 26294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:35.651180 26294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:35.651288 26294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:35.651361 26294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:11:35.661576 26294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:45125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:45125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:35.662920 26294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:35.664326 26294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:35.676229 26304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:35.676151 26307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:35.676187 26309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:35.677050 26294 server_base.cc:1034] running on GCE node
I20260812 01:11:35.677855 26294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:35.679247 26294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:35.680495 26294 hybrid_clock.cc:648] HybridClock initialized: now 1786497095680446 us; error 40 us; skew 500 ppm
I20260812 01:11:35.680859 26294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:35 dist-test-slave-21b9 krb5kdc[26273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497095, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:35.688613 26294 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:11:35.690233 26294 webserver.cc:460] Webserver started at http://127.20.151.62:42163/ using document root <none> and password file <none>
I20260812 01:11:35.690968 26294 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:35.691095 26294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:35.691516 26294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:35.694977 26294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "9b9a32a179ff451a95dde8dadfe21f5a"
format_stamp: "Formatted at 2026-08-12 01:11:35 on dist-test-slave-21b9"
server_key: "ee55b0de27dc3a6528c717f3df3c9cd1"
server_key_iv: "06ce669b9a4022c2d86d56052eb1e3e6"
server_key_version: "encryptionkey@0"
I20260812 01:11:35.695856 26294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "9b9a32a179ff451a95dde8dadfe21f5a"
format_stamp: "Formatted at 2026-08-12 01:11:35 on dist-test-slave-21b9"
server_key: "ee55b0de27dc3a6528c717f3df3c9cd1"
server_key_iv: "06ce669b9a4022c2d86d56052eb1e3e6"
server_key_version: "encryptionkey@0"
I20260812 01:11:35.701311 26294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:11:35.705046 26325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:35.706493 26294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:11:35.706688 26294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "9b9a32a179ff451a95dde8dadfe21f5a"
format_stamp: "Formatted at 2026-08-12 01:11:35 on dist-test-slave-21b9"
server_key: "ee55b0de27dc3a6528c717f3df3c9cd1"
server_key_iv: "06ce669b9a4022c2d86d56052eb1e3e6"
server_key_version: "encryptionkey@0"
I20260812 01:11:35.706897 26294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:35.720285 26294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:35.725111 26294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:35.725421 26294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:35.744777 26294 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:45125
I20260812 01:11:35.744819 26402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:45125 every 8 connection(s)
I20260812 01:11:35.746266 26294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:35.750240 26403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:35.755674 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 26294
I20260812 01:11:35.755901 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:35.756546 21084 external_mini_cluster.cc:1428] Setting key c47f9af40df6104f02ed3dd9f516b6fb
I20260812 01:11:35.761554 26403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a: Bootstrap starting.
I20260812 01:11:35.765343 26403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:35.766404 26403 log.cc:824] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:35.769593 26403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a: No bootstrap required, opened a new log
Aug 12 01:11:35 dist-test-slave-21b9 krb5kdc[26273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:35.780143 26403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a32a179ff451a95dde8dadfe21f5a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45125 } }
I20260812 01:11:35.780656 26403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:35.780776 26403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9a32a179ff451a95dde8dadfe21f5a, State: Initialized, Role: FOLLOWER
I20260812 01:11:35.781522 26403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a32a179ff451a95dde8dadfe21f5a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45125 } }
I20260812 01:11:35.781836 26403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:35.781860 26408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:35.758445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.20.151.62:45125)
0812 01:11:35.759455 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:35.759467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:35.759523 (+ 56us) server_negotiation.cc:407] Connection header received
0812 01:11:35.761282 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:35.761305 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:35.761712 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:35.762068 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:35.764028 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:35.765348 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:35.766687 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:35.767630 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:35.767777 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:35.772301 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:35.772314 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:35.772336 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:35.772384 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:35.776771 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:35.777669 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:35.777680 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:35.777694 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:35.777944 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:35.778317 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:35.778321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:35.778325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:35.779183 (+ 858us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:35.779508 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:35.779933 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 01:11:35.780441 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":410,"threads_started":1}
I20260812 01:11:35.782020 26403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:35.782213 26403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:35.784799 26403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a32a179ff451a95dde8dadfe21f5a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45125 } }
I20260812 01:11:35.785328 26403 leader_election.cc:302] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b9a32a179ff451a95dde8dadfe21f5a; no voters:
I20260812 01:11:35.786006 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:11:35.785969 26403 leader_election.cc:288] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:35.786055 26410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:35.787169 26410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [term 1 LEADER]: Becoming Leader. State: Replica: 9b9a32a179ff451a95dde8dadfe21f5a, State: Running, Role: LEADER
I20260812 01:11:35.787696 26410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a32a179ff451a95dde8dadfe21f5a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45125 } }
I20260812 01:11:35.788249 26403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:35.790463 26414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b9a32a179ff451a95dde8dadfe21f5a. Latest consensus state: current_term: 1 leader_uuid: "9b9a32a179ff451a95dde8dadfe21f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a32a179ff451a95dde8dadfe21f5a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45125 } } }
I20260812 01:11:35.790881 26414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:35.790446 26412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b9a32a179ff451a95dde8dadfe21f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9a32a179ff451a95dde8dadfe21f5a" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45125 } } }
I20260812 01:11:35.791428 26412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:35.791795 26419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:35.810039 26419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:35.829327 26419 catalog_manager.cc:1347] Generated new cluster ID: edeaf9eea9884abb92b80db39b9a49f2
I20260812 01:11:35.829469 26419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:35.852051 26408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:35.844237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.20.151.62:45125)
0812 01:11:35.844482 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:35.844490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:35.845087 (+ 597us) server_negotiation.cc:407] Connection header received
0812 01:11:35.845280 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:35.845285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:35.845360 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:35.845448 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:35.846576 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:35.847362 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:35.848701 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:35.849174 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:35.849293 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:35.850444 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:35.850449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:35.850453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:35.851135 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:35.851207 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:35.851238 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:35.851365 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:35.851564 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 01:11:35.851673 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:11:35.854594 26419 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:35.856480 26419 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:35.863617 26419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b9a32a179ff451a95dde8dadfe21f5a: Generated new TSK 0
I20260812 01:11:35.864691 26419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:11:35.873608 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26294
I20260812 01:11:35.874970 26398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:35.968345 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 26294
2026-08-12T01:11:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2170 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:11:36 dist-test-slave-21b9 krb5kdc[26458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:11:36 dist-test-slave-21b9 krb5kdc[26458](info): set up 2 sockets
Aug 12 01:11:36 dist-test-slave-21b9 krb5kdc[26458](info): commencing operation
krb5kdc: starting...
W20260812 01:11:37.223794 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:11:37 dist-test-slave-21b9 krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:11:37.470319 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:41065
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:41065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:37.809010 26475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:37.809366 26475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:37.809507 26475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:37.816881 26475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:37.816998 26475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:37.817072 26475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:37.817150 26475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:37.817212 26475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:11:37.827231 26475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:41065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:41065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:37.828629 26475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:37.829941 26475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:37.841818 26485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:37.841866 26488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:37.841957 26486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:37.843010 26475 server_base.cc:1034] running on GCE node
I20260812 01:11:37.843900 26475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:37.845397 26475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:37.846655 26475 hybrid_clock.cc:648] HybridClock initialized: now 1786497097846603 us; error 29 us; skew 500 ppm
I20260812 01:11:37.847043 26475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:37 dist-test-slave-21b9 krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497097, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:37.850356 26475 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:11:37.852065 26475 webserver.cc:460] Webserver started at http://127.20.151.62:46685/ using document root <none> and password file <none>
I20260812 01:11:37.852777 26475 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:37.852905 26475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:37.853214 26475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:37.856544 26475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "555a3aa6d53f4abe94ca089db1d76190"
format_stamp: "Formatted at 2026-08-12 01:11:37 on dist-test-slave-21b9"
server_key: "6b47f341d7c79fdb42c12801c65c088c"
server_key_iv: "30a20301d5023903485c5450505d24a4"
server_key_version: "encryptionkey@0"
I20260812 01:11:37.857334 26475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "555a3aa6d53f4abe94ca089db1d76190"
format_stamp: "Formatted at 2026-08-12 01:11:37 on dist-test-slave-21b9"
server_key: "6b47f341d7c79fdb42c12801c65c088c"
server_key_iv: "30a20301d5023903485c5450505d24a4"
server_key_version: "encryptionkey@0"
I20260812 01:11:37.862704 26475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:11:37.866379 26503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:37.867832 26475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:11:37.868018 26475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "555a3aa6d53f4abe94ca089db1d76190"
format_stamp: "Formatted at 2026-08-12 01:11:37 on dist-test-slave-21b9"
server_key: "6b47f341d7c79fdb42c12801c65c088c"
server_key_iv: "30a20301d5023903485c5450505d24a4"
server_key_version: "encryptionkey@0"
I20260812 01:11:37.868220 26475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:37.888803 26475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:37.892953 26475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:37.893249 26475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:37.911208 26475 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:41065
I20260812 01:11:37.911247 26591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:41065 every 8 connection(s)
I20260812 01:11:37.912674 26475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:37.915207 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 26475
I20260812 01:11:37.915442 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:37.916138 21084 external_mini_cluster.cc:1428] Setting key 416dd96bfdedb5f168eb022bec7622a6
I20260812 01:11:37.916682 26593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:37.931391 26593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190: Bootstrap starting.
Aug 12 01:11:37 dist-test-slave-21b9 krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:37.934919 26593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:37.936077 26593 log.cc:824] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:37.938995 26593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190: No bootstrap required, opened a new log
I20260812 01:11:37.940070 26599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:37.918118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.20.151.62:41065)
0812 01:11:37.918911 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:37.918923 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:37.918968 (+ 45us) server_negotiation.cc:407] Connection header received
0812 01:11:37.921380 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:37.921405 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:37.921871 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:37.922227 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:37.924115 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:37.925428 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:37.927130 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:37.927964 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:37.928127 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:37.932833 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:37.932842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:37.932860 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:37.932910 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:37.935998 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:37.936716 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:37.936722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:37.936729 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:37.936913 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:37.937322 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:37.937326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:37.937328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:37.937863 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:37.938176 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:37.938604 (+ 428us) server_negotiation.cc:299] Negotiation successful
0812 01:11:37.939036 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":366,"threads_started":1}
I20260812 01:11:37.947237 26593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a3aa6d53f4abe94ca089db1d76190" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41065 } }
I20260812 01:11:37.947762 26593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:37.947873 26593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555a3aa6d53f4abe94ca089db1d76190, State: Initialized, Role: FOLLOWER
I20260812 01:11:37.948491 26593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a3aa6d53f4abe94ca089db1d76190" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41065 } }
I20260812 01:11:37.948721 26593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:37.948851 26593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:37.949008 26593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:37.951213 26593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a3aa6d53f4abe94ca089db1d76190" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41065 } }
I20260812 01:11:37.951752 26593 leader_election.cc:302] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 555a3aa6d53f4abe94ca089db1d76190; no voters:
I20260812 01:11:37.952329 26593 leader_election.cc:288] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:37.952461 26604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:37.953133 26604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [term 1 LEADER]: Becoming Leader. State: Replica: 555a3aa6d53f4abe94ca089db1d76190, State: Running, Role: LEADER
I20260812 01:11:37.953720 26604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a3aa6d53f4abe94ca089db1d76190" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41065 } }
I20260812 01:11:37.954347 26593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:37.956928 26605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "555a3aa6d53f4abe94ca089db1d76190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a3aa6d53f4abe94ca089db1d76190" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41065 } } }
I20260812 01:11:37.957000 26606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 555a3aa6d53f4abe94ca089db1d76190. Latest consensus state: current_term: 1 leader_uuid: "555a3aa6d53f4abe94ca089db1d76190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a3aa6d53f4abe94ca089db1d76190" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41065 } } }
I20260812 01:11:37.957381 26606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:37.957376 26605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:37.959193 26614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:37.965096 26614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:37.974141 26614 catalog_manager.cc:1347] Generated new cluster ID: 8143d38f02e44a6db61cd9c47c038785
I20260812 01:11:37.974267 26614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:37.995214 26614 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:37.996636 26614 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:38.005558 26614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 555a3aa6d53f4abe94ca089db1d76190: Generated new TSK 0
I20260812 01:11:38.006793 26614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:11:38.071486 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:41065
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:38.428349 26645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:38.428658 26645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:38.428807 26645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:38.436146 26645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:38.436281 26645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:38.436436 26645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:11:38.446710 26645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:41065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:38.448079 26645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:38.449470 26645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:38.463012 26658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:38.463137 26655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:38.463114 26660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:38.463747 26645 server_base.cc:1034] running on GCE node
I20260812 01:11:38.464440 26645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:38.465444 26645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:38.466701 26645 hybrid_clock.cc:648] HybridClock initialized: now 1786497098466636 us; error 55 us; skew 500 ppm
I20260812 01:11:38.467038 26645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:38 dist-test-slave-21b9 krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497098, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:38.470302 26645 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:11:38.471909 26645 webserver.cc:460] Webserver started at http://127.20.151.1:34965/ using document root <none> and password file <none>
I20260812 01:11:38.472657 26645 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:38.472785 26645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:38.473132 26645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:38.476282 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "68b90ea38b0047e18b21dc463de3d8f4"
format_stamp: "Formatted at 2026-08-12 01:11:38 on dist-test-slave-21b9"
server_key: "d0760bf76677c9acf6c8dd493e63f438"
server_key_iv: "fd45e848d5131770d5d54e356174bf82"
server_key_version: "encryptionkey@0"
I20260812 01:11:38.477074 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "68b90ea38b0047e18b21dc463de3d8f4"
format_stamp: "Formatted at 2026-08-12 01:11:38 on dist-test-slave-21b9"
server_key: "d0760bf76677c9acf6c8dd493e63f438"
server_key_iv: "fd45e848d5131770d5d54e356174bf82"
server_key_version: "encryptionkey@0"
I20260812 01:11:38.482342 26645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 01:11:38.485862 26674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:38.487182 26645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:11:38.487363 26645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "68b90ea38b0047e18b21dc463de3d8f4"
format_stamp: "Formatted at 2026-08-12 01:11:38 on dist-test-slave-21b9"
server_key: "d0760bf76677c9acf6c8dd493e63f438"
server_key_iv: "fd45e848d5131770d5d54e356174bf82"
server_key_version: "encryptionkey@0"
I20260812 01:11:38.487555 26645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:38.514185 26645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:38.518419 26645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:38.518726 26645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:38.519783 26645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:38.521214 26645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:38.521296 26645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:38.521422 26645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:38.521477 26645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:38.554221 26645 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:35579
I20260812 01:11:38.554301 26867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:35579 every 8 connection(s)
I20260812 01:11:38.555745 26645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:11:38.560866 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 26645
I20260812 01:11:38.561120 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:11:38.561987 21084 external_mini_cluster.cc:1428] Setting key fa5c21dd4c5de386dce2f7631449de12
Aug 12 01:11:38 dist-test-slave-21b9 krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497098, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:38.578454 26870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:38.558450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:50859 (local address 127.20.151.62:41065)
0812 01:11:38.558909 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:38.558914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:38.559858 (+ 944us) server_negotiation.cc:407] Connection header received
0812 01:11:38.562608 (+ 2750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:38.562613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:38.562697 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:38.562806 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:38.564901 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:38.565883 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:38.567599 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:38.568156 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:38.568287 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:38.571989 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:38.572000 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:38.572004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:38.572045 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:38.574816 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:38.575826 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:38.575831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:38.575833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:38.575986 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:38.576520 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:38.576524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:38.576526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:38.576829 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:38.576972 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:38.577815 (+ 843us) server_negotiation.cc:299] Negotiation successful
0812 01:11:38.578084 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":245,"threads_started":1}
I20260812 01:11:38.579607 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:38.558858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:41065 (local address 127.20.151.1:50859)
0812 01:11:38.559658 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:38.559719 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:38.562226 (+ 2507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:38.563359 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:38.563374 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:38.564036 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:38.564707 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:38.564727 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:38.566045 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:38.566050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:38.567421 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:38.567435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:38.568406 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:38.568411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:38.568642 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:38.569345 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:38.569369 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:38.571721 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:38.574988 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:38.574999 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:38.575018 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:38.575675 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:38.576153 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:38.576157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:38.576159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:38.576388 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:38.576974 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:38.576983 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:38.577399 (+ 416us) client_negotiation.cc:769] Sending connection context
0812 01:11:38.577668 (+ 269us) client_negotiation.cc:240] Negotiation successful
0812 01:11:38.578284 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":323,"threads_started":1}
I20260812 01:11:38.581331 26868 heartbeater.cc:347] Connected to a master server at 127.20.151.62:41065
I20260812 01:11:38.581759 26868 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:38.582731 26868 heartbeater.cc:511] Master 127.20.151.62:41065 requested a full tablet report, sending...
I20260812 01:11:38.585305 26532 ts_manager.cc:194] Registered new tserver with Master: 68b90ea38b0047e18b21dc463de3d8f4 (127.20.151.1:35579)
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
I20260812 01:11:38.586833 26532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:50859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:11:38.618896 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:41065
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:38.962916 26884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:38.963223 26884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:38.963395 26884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:38.970709 26884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:38.970853 26884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:38.971017 26884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:11:38.981179 26884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:41065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:38.982589 26884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:38.984045 26884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:38.998059 26894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:38.998027 26893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:38.998271 26898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:38.998780 26884 server_base.cc:1034] running on GCE node
I20260812 01:11:38.999460 26884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:39.000500 26884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:39.001705 26884 hybrid_clock.cc:648] HybridClock initialized: now 1786497099001645 us; error 46 us; skew 500 ppm
I20260812 01:11:39.002018 26884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:38 dist-test-slave-21b9 krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497098, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:39.005283 26884 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:11:39.006968 26884 webserver.cc:460] Webserver started at http://127.20.151.2:40931/ using document root <none> and password file <none>
I20260812 01:11:39.007727 26884 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:39.007848 26884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:39.008155 26884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:39.011245 26884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "ef850942d7094289a79061b37bdc986c"
format_stamp: "Formatted at 2026-08-12 01:11:39 on dist-test-slave-21b9"
server_key: "44ce4fe2e2e7b6ae43380b6cbda9386b"
server_key_iv: "039b3cba14049812ff110927eb5655af"
server_key_version: "encryptionkey@0"
I20260812 01:11:39.012089 26884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "ef850942d7094289a79061b37bdc986c"
format_stamp: "Formatted at 2026-08-12 01:11:39 on dist-test-slave-21b9"
server_key: "44ce4fe2e2e7b6ae43380b6cbda9386b"
server_key_iv: "039b3cba14049812ff110927eb5655af"
server_key_version: "encryptionkey@0"
I20260812 01:11:39.017427 26884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:11:39.021133 26908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:39.022642 26884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:11:39.022823 26884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "ef850942d7094289a79061b37bdc986c"
format_stamp: "Formatted at 2026-08-12 01:11:39 on dist-test-slave-21b9"
server_key: "44ce4fe2e2e7b6ae43380b6cbda9386b"
server_key_iv: "039b3cba14049812ff110927eb5655af"
server_key_version: "encryptionkey@0"
I20260812 01:11:39.023036 26884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:39.043136 26884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:39.048028 26884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:39.048372 26884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:39.049468 26884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:39.050994 26884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:39.051092 26884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:39.051244 26884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:39.051323 26884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 01:11:39.085026 26884 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:37577
I20260812 01:11:39.085104 27106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:37577 every 8 connection(s)
I20260812 01:11:39.086360 26884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:11:39.093544 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 26884
I20260812 01:11:39.093724 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:11:39.094200 21084 external_mini_cluster.cc:1428] Setting key 6ee465c8c8cd9c846912214697831241
Aug 12 01:11:39 dist-test-slave-21b9 krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497098, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:39.106472 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:39.089093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:34415 (local address 127.20.151.62:41065)
0812 01:11:39.089604 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:39.089609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:39.090472 (+ 863us) server_negotiation.cc:407] Connection header received
0812 01:11:39.092442 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:39.092448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:39.092535 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:39.092643 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:39.094333 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.095105 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.096756 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.097194 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.097305 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:39.100809 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:39.100814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:39.100817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:39.100851 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:39.103389 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.104134 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.104138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.104141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.104271 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.104696 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.104700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.104702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.104973 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:39.105106 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:39.105837 (+ 731us) server_negotiation.cc:299] Negotiation successful
0812 01:11:39.106077 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":256,"threads_started":1}
I20260812 01:11:39.107522 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:39.089528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:41065 (local address 127.20.151.2:34415)
0812 01:11:39.090301 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:39.090343 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:39.092125 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:39.092908 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:39.092917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:39.093465 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:39.094150 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:39.094170 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.095297 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.095303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:39.096612 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:39.096625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.097431 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.097438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:39.097668 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:39.098325 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:39.098354 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:39.100579 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:39.103532 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:39.103541 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:39.103559 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:39.104001 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:39.104377 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:39.104382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:39.104385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:39.104579 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:39.105105 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:39.105113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:39.105461 (+ 348us) client_negotiation.cc:769] Sending connection context
0812 01:11:39.105712 (+ 251us) client_negotiation.cc:240] Negotiation successful
0812 01:11:39.106291 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":284,"threads_started":1}
I20260812 01:11:39.109196 27107 heartbeater.cc:347] Connected to a master server at 127.20.151.62:41065
I20260812 01:11:39.109561 27107 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:39.110364 27107 heartbeater.cc:511] Master 127.20.151.62:41065 requested a full tablet report, sending...
I20260812 01:11:39.112104 26532 ts_manager.cc:194] Registered new tserver with Master: ef850942d7094289a79061b37bdc986c (127.20.151.2:37577)
I20260812 01:11:39.113168 26532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:34415
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:11:39.148918 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:41065
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:39.484526 27116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:39.484812 27116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:39.484951 27116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:39.492995 27116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:39.493113 27116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:39.493266 27116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:11:39.503219 27116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:41065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:39.504642 27116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:39.506306 27116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:39.519744 27127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:39.519680 27125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:39.519966 27123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:39.520608 27116 server_base.cc:1034] running on GCE node
I20260812 01:11:39.521299 27116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:39.522295 27116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:39.523517 27116 hybrid_clock.cc:648] HybridClock initialized: now 1786497099523454 us; error 47 us; skew 500 ppm
I20260812 01:11:39.523854 27116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:39 dist-test-slave-21b9 krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497099, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:39.527244 27116 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:11:39.528986 27116 webserver.cc:460] Webserver started at http://127.20.151.3:42975/ using document root <none> and password file <none>
I20260812 01:11:39.529771 27116 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:39.529908 27116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:39.530237 27116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:39.533506 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "bbd64e5002ef4d8497a9eea7bb3fe8b2"
format_stamp: "Formatted at 2026-08-12 01:11:39 on dist-test-slave-21b9"
server_key: "818496804cd07d47d4189006d43bd57f"
server_key_iv: "fc25c35e3211e999495f79ee65aa0a16"
server_key_version: "encryptionkey@0"
I20260812 01:11:39.534394 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "bbd64e5002ef4d8497a9eea7bb3fe8b2"
format_stamp: "Formatted at 2026-08-12 01:11:39 on dist-test-slave-21b9"
server_key: "818496804cd07d47d4189006d43bd57f"
server_key_iv: "fc25c35e3211e999495f79ee65aa0a16"
server_key_version: "encryptionkey@0"
I20260812 01:11:39.540441 27116 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 01:11:39.544587 27135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:39.546133 27116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260812 01:11:39.546319 27116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "bbd64e5002ef4d8497a9eea7bb3fe8b2"
format_stamp: "Formatted at 2026-08-12 01:11:39 on dist-test-slave-21b9"
server_key: "818496804cd07d47d4189006d43bd57f"
server_key_iv: "fc25c35e3211e999495f79ee65aa0a16"
server_key_version: "encryptionkey@0"
I20260812 01:11:39.546547 27116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:39.570104 27116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:39.574800 27116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:39.575098 27116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:39.576210 27116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:39.577654 27116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:39.577744 27116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:39.577870 27116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:39.577925 27116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:39.590322 26868 heartbeater.cc:503] Master 127.20.151.62:41065 was elected leader, sending a full tablet report...
I20260812 01:11:39.612571 27116 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:44275
I20260812 01:11:39.612666 27329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:44275 every 8 connection(s)
I20260812 01:11:39.613972 27116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:11:39.624006 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 27116
I20260812 01:11:39.624207 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:11:39.624689 21084 external_mini_cluster.cc:1428] Setting key abaebcaa66fa576dfe32ba2cfe11ff55
Aug 12 01:11:39 dist-test-slave-21b9 krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497099, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:39.634088 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:39.616498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:43903 (local address 127.20.151.62:41065)
0812 01:11:39.616965 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:39.616970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:39.617951 (+ 981us) server_negotiation.cc:407] Connection header received
0812 01:11:39.619829 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:39.619836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:39.619918 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:39.620027 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:39.621557 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.622371 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.624219 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.624725 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.624865 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:39.628829 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:39.628834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:39.628837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:39.628877 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:39.631327 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.632000 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.632003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.632006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.632127 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.632472 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.632476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.632478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.632809 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:39.632930 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:39.633623 (+ 693us) server_negotiation.cc:299] Negotiation successful
0812 01:11:39.633804 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":241,"threads_started":1}
I20260812 01:11:39.635032 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:39.616934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:41065 (local address 127.20.151.3:43903)
0812 01:11:39.617767 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:39.617815 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:39.619561 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:39.620285 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:39.620294 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:39.620822 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:39.621375 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:39.621393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.622531 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.622535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:39.624049 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:39.624065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.624976 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.624985 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:39.625239 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:39.626055 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:39.626100 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:39.628612 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:39.631453 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:39.631460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:39.631473 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:39.631905 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:39.632205 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:39.632208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:39.632210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:39.632394 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:39.632904 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:39.632910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:39.633264 (+ 354us) client_negotiation.cc:769] Sending connection context
0812 01:11:39.633499 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 01:11:39.633922 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":320,"threads_started":1}
I20260812 01:11:39.636595 27330 heartbeater.cc:347] Connected to a master server at 127.20.151.62:41065
I20260812 01:11:39.636904 27330 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:39.637652 27330 heartbeater.cc:511] Master 127.20.151.62:41065 requested a full tablet report, sending...
I20260812 01:11:39.639348 26531 ts_manager.cc:194] Registered new tserver with Master: bbd64e5002ef4d8497a9eea7bb3fe8b2 (127.20.151.3:44275)
I20260812 01:11:39.640580 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:11:39.640609 26531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:43903
Aug 12 01:11:39 dist-test-slave-21b9 krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497099, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 01:11:39 dist-test-slave-21b9 krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497099, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:39.699956 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:39.686155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.20.151.62:41065)
0812 01:11:39.686408 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:39.686415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:39.686779 (+ 364us) server_negotiation.cc:407] Connection header received
0812 01:11:39.686979 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:39.686983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:39.687057 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:39.687148 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:39.688636 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.689409 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.690662 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.691116 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.691232 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:39.695142 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:39.695148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:39.695151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:39.695189 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:39.697727 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.698337 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.698340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.698343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.698464 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.698764 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.698767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.698769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.699083 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:39.699260 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:39.699526 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 01:11:39.699711 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260812 01:11:39.700536 26531 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54532
I20260812 01:11:39.702127 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26645
Aug 12 01:11:39 dist-test-slave-21b9 krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497099, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:39.717826 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:39.703084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.20.151.1:35579)
0812 01:11:39.703689 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:39.703697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:39.703721 (+ 24us) server_negotiation.cc:407] Connection header received
0812 01:11:39.703780 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:39.703786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:39.703991 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:39.704137 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:39.705297 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.706336 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.707657 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.708259 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.708357 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:39.712260 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:39.712274 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:39.712290 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:39.712325 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:39.715203 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.715899 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.715914 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.715916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.716058 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.716362 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.716365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.716367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.716726 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:39.716963 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:39.717286 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 01:11:39.717496 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":273,"threads_started":1}
W20260812 01:11:39.719198 26860 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52154
W20260812 01:11:39.720356 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26645: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:11:39.720578 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 26645
I20260812 01:11:39.756392 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26884
Aug 12 01:11:39 dist-test-slave-21b9 krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497099, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:11:39.772635 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:39.757352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.20.151.2:37577)
0812 01:11:39.757906 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:39.757914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:39.757941 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:11:39.758010 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:39.758016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:39.758237 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:39.758373 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:39.759643 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.760763 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.762026 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.762599 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.762701 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:39.766808 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:39.766821 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:39.766838 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:39.766878 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:39.769846 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.770445 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.770451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.770454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.770578 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.770842 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.770845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.770847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.771209 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:39.771573 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:39.771976 (+ 403us) server_negotiation.cc:299] Negotiation successful
0812 01:11:39.772276 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":286,"threads_started":1}
W20260812 01:11:39.773922 27095 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58800
W20260812 01:11:39.775100 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26884: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:11:39.775259 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 26884
I20260812 01:11:39.802186 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27116
Aug 12 01:11:39 dist-test-slave-21b9 krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497099, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:11:39.819047 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:39.803242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42568 (local address 127.20.151.3:44275)
0812 01:11:39.803925 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:39.803933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:39.803972 (+ 39us) server_negotiation.cc:407] Connection header received
0812 01:11:39.804036 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:39.804043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:39.804265 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:39.804416 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:39.805879 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.807000 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.808355 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:39.808952 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:39.809066 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:39.813386 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:39.813399 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:39.813414 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:39.813456 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:39.816245 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.816919 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.816926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.816928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.817069 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:39.817427 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:39.817431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:39.817433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:39.817800 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:39.818123 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:39.818414 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 01:11:39.818693 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":334,"threads_started":1}
W20260812 01:11:39.820523 27324 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42568
W20260812 01:11:39.821676 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27116: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:11:39.821851 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 27116
I20260812 01:11:39.845795 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26475
I20260812 01:11:39.847122 26586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:39.954746 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 26475
2026-08-12T01:11:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3990 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:11:40 dist-test-slave-21b9 krb5kdc[27357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:11:40 dist-test-slave-21b9 krb5kdc[27357](info): set up 2 sockets
Aug 12 01:11:40 dist-test-slave-21b9 krb5kdc[27357](info): commencing operation
krb5kdc: starting...
W20260812 01:11:41.211825 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.009s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:11:41 dist-test-slave-21b9 krb5kdc[27357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:11:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:11:41.463912 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:35273
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:35273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:41.807973 27379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:41.808298 27379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:41.808441 27379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:41.815899 27379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:41.816035 27379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:41.816134 27379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:41.816219 27379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:41.816283 27379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:11:41.826308 27379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:35273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:35273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:41.827711 27379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:41.829020 27379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:41.840852 27386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:41.840821 27391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:41.840927 27388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:41.841763 27379 server_base.cc:1034] running on GCE node
I20260812 01:11:41.842562 27379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:41.844046 27379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:41.845258 27379 hybrid_clock.cc:648] HybridClock initialized: now 1786497101845202 us; error 46 us; skew 500 ppm
I20260812 01:11:41.845613 27379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:41 dist-test-slave-21b9 krb5kdc[27357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497101, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:41.852005 27379 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:11:41.853574 27379 webserver.cc:460] Webserver started at http://127.20.151.62:43545/ using document root <none> and password file <none>
I20260812 01:11:41.854287 27379 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:41.854395 27379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:41.854666 27379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:41.857790 27379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "41b6654ae6fe48cdb9dc03a96c321634"
format_stamp: "Formatted at 2026-08-12 01:11:41 on dist-test-slave-21b9"
server_key: "2b71e862c0fc33536fbe92c1bf5f6f29"
server_key_iv: "977b7ee3242be4e2429f4d81c70e4dd2"
server_key_version: "encryptionkey@0"
I20260812 01:11:41.858608 27379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "41b6654ae6fe48cdb9dc03a96c321634"
format_stamp: "Formatted at 2026-08-12 01:11:41 on dist-test-slave-21b9"
server_key: "2b71e862c0fc33536fbe92c1bf5f6f29"
server_key_iv: "977b7ee3242be4e2429f4d81c70e4dd2"
server_key_version: "encryptionkey@0"
I20260812 01:11:41.863984 27379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:11:41.867960 27405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:41.869341 27379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:41.869537 27379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "41b6654ae6fe48cdb9dc03a96c321634"
format_stamp: "Formatted at 2026-08-12 01:11:41 on dist-test-slave-21b9"
server_key: "2b71e862c0fc33536fbe92c1bf5f6f29"
server_key_iv: "977b7ee3242be4e2429f4d81c70e4dd2"
server_key_version: "encryptionkey@0"
I20260812 01:11:41.869733 27379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:41.900300 27379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:41.904345 27379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:41.904636 27379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:41.923359 27379 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:35273
I20260812 01:11:41.923409 27497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:35273 every 8 connection(s)
I20260812 01:11:41.924790 27379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:41.928685 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:41.929842 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 27379
I20260812 01:11:41.930099 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:41.930922 21084 external_mini_cluster.cc:1428] Setting key 015bc248ead619794594b8eb95754503
I20260812 01:11:41.943780 27498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634: Bootstrap starting.
Aug 12 01:11:41 dist-test-slave-21b9 krb5kdc[27357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:41.947619 27498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:41.948822 27498 log.cc:824] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:41.952353 27498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634: No bootstrap required, opened a new log
I20260812 01:11:41.955150 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:41.933079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.20.151.62:35273)
0812 01:11:41.934102 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:41.934114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:41.934156 (+ 42us) server_negotiation.cc:407] Connection header received
0812 01:11:41.935877 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:41.935904 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:41.936315 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:41.936682 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:41.938537 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:41.939894 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:41.941671 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:41.942336 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:41.942471 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:41.947289 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:41.947301 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:41.947321 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:41.947367 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:41.950884 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:41.951791 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:41.951798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:41.951804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:41.951979 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:41.952406 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:41.952411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:41.952414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:41.952994 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:41.953259 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:41.953694 (+ 435us) server_negotiation.cc:299] Negotiation successful
0812 01:11:41.954031 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":539,"threads_started":1}
I20260812 01:11:41.960502 27498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b6654ae6fe48cdb9dc03a96c321634" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35273 } }
I20260812 01:11:41.960872 27498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:41.960984 27498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b6654ae6fe48cdb9dc03a96c321634, State: Initialized, Role: FOLLOWER
I20260812 01:11:41.961511 27498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b6654ae6fe48cdb9dc03a96c321634" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35273 } }
I20260812 01:11:41.961751 27498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:41.961879 27498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:41.962033 27498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:41.964285 27498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b6654ae6fe48cdb9dc03a96c321634" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35273 } }
I20260812 01:11:41.964766 27498 leader_election.cc:302] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41b6654ae6fe48cdb9dc03a96c321634; no voters:
I20260812 01:11:41.965337 27498 leader_election.cc:288] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:41.965457 27506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:41.966172 27506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [term 1 LEADER]: Becoming Leader. State: Replica: 41b6654ae6fe48cdb9dc03a96c321634, State: Running, Role: LEADER
I20260812 01:11:41.966738 27506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b6654ae6fe48cdb9dc03a96c321634" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35273 } }
I20260812 01:11:41.967434 27498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:41.969292 27508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41b6654ae6fe48cdb9dc03a96c321634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b6654ae6fe48cdb9dc03a96c321634" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35273 } } }
I20260812 01:11:41.969586 27509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41b6654ae6fe48cdb9dc03a96c321634. Latest consensus state: current_term: 1 leader_uuid: "41b6654ae6fe48cdb9dc03a96c321634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b6654ae6fe48cdb9dc03a96c321634" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35273 } } }
I20260812 01:11:41.969786 27508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:41.969869 27509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:41.980317 27512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:41.985634 27512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:41.995438 27512 catalog_manager.cc:1347] Generated new cluster ID: 984d6ef5f53c4b91bc4687b108b140b1
I20260812 01:11:41.995571 27512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:42.017220 27512 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:42.018831 27512 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:42.034941 27512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41b6654ae6fe48cdb9dc03a96c321634: Generated new TSK 0
I20260812 01:11:42.036147 27512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:11:42.095185 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35273
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:42.443807 27546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:42.444116 27546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:42.444216 27546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:42.451711 27546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:42.451838 27546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:42.451967 27546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:11:42.462128 27546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:42.463496 27546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:42.464840 27546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:42.478631 27556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:42.478605 27558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:42.478578 27555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:42.479948 27546 server_base.cc:1034] running on GCE node
I20260812 01:11:42.480701 27546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:42.481781 27546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:42.483047 27546 hybrid_clock.cc:648] HybridClock initialized: now 1786497102482975 us; error 58 us; skew 500 ppm
I20260812 01:11:42.483439 27546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:42 dist-test-slave-21b9 krb5kdc[27357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497102, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:42.486888 27546 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:11:42.488590 27546 webserver.cc:460] Webserver started at http://127.20.151.1:41157/ using document root <none> and password file <none>
I20260812 01:11:42.489331 27546 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:42.489485 27546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:42.489842 27546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:42.492954 27546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "c1e014cc1a9341bba0f101ee6bc37c5b"
format_stamp: "Formatted at 2026-08-12 01:11:42 on dist-test-slave-21b9"
server_key: "10ab47e60888f349802b3d53fe9d7d09"
server_key_iv: "d34b9dc22c037c7701b73d5c7ea78668"
server_key_version: "encryptionkey@0"
I20260812 01:11:42.493731 27546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "c1e014cc1a9341bba0f101ee6bc37c5b"
format_stamp: "Formatted at 2026-08-12 01:11:42 on dist-test-slave-21b9"
server_key: "10ab47e60888f349802b3d53fe9d7d09"
server_key_iv: "d34b9dc22c037c7701b73d5c7ea78668"
server_key_version: "encryptionkey@0"
I20260812 01:11:42.499065 27546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:11:42.502840 27569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:42.504293 27546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:11:42.504475 27546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "c1e014cc1a9341bba0f101ee6bc37c5b"
format_stamp: "Formatted at 2026-08-12 01:11:42 on dist-test-slave-21b9"
server_key: "10ab47e60888f349802b3d53fe9d7d09"
server_key_iv: "d34b9dc22c037c7701b73d5c7ea78668"
server_key_version: "encryptionkey@0"
I20260812 01:11:42.504683 27546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:42.531073 27546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:42.535846 27546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:42.536158 27546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:42.537169 27546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:42.538604 27546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:42.538684 27546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:42.538808 27546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:42.538863 27546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:42.572011 27546 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:45673
I20260812 01:11:42.572124 27764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:45673 every 8 connection(s)
I20260812 01:11:42.573468 27546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:11:42.580996 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 27546
I20260812 01:11:42.581246 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:11:42.582084 21084 external_mini_cluster.cc:1428] Setting key 3a816dcc22a2d963aa011779d4b75723
Aug 12 01:11:42 dist-test-slave-21b9 krb5kdc[27357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497102, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:42.595937 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:42.576659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:42459 (local address 127.20.151.62:35273)
0812 01:11:42.577207 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:42.577214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:42.578135 (+ 921us) server_negotiation.cc:407] Connection header received
0812 01:11:42.580076 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:42.580080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:42.580150 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:42.580237 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:42.582233 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:42.583116 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:42.585084 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:42.585648 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:42.585777 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:42.589532 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:42.589538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:42.589542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:42.589584 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:42.592699 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:42.593477 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:42.593482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:42.593485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:42.593628 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:42.594053 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:42.594057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:42.594060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:42.594370 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:42.594484 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:42.595468 (+ 984us) server_negotiation.cc:299] Negotiation successful
0812 01:11:42.595677 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":297,"threads_started":1}
I20260812 01:11:42.597384 27771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:42.577251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:35273 (local address 127.20.151.1:42459)
0812 01:11:42.577980 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:42.578021 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:42.579807 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:42.580415 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:42.580426 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:42.581190 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:42.582017 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:42.582044 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:42.583299 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:42.583304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:42.584918 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:42.584936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:42.585881 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:42.585886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:42.586119 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:42.586793 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:42.586815 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:42.589298 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:42.592855 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:42.592863 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:42.592877 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:42.593349 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:42.593722 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:42.593725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:42.593727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:42.593936 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:42.594511 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:42.594519 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:42.594995 (+ 476us) client_negotiation.cc:769] Sending connection context
0812 01:11:42.595367 (+ 372us) client_negotiation.cc:240] Negotiation successful
0812 01:11:42.596002 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":264,"threads_started":1}
I20260812 01:11:42.599293 27766 heartbeater.cc:347] Connected to a master server at 127.20.151.62:35273
I20260812 01:11:42.599622 27766 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:42.600354 27766 heartbeater.cc:511] Master 127.20.151.62:35273 requested a full tablet report, sending...
I20260812 01:11:42.602728 27436 ts_manager.cc:194] Registered new tserver with Master: c1e014cc1a9341bba0f101ee6bc37c5b (127.20.151.1:45673)
I20260812 01:11:42.604821 27436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:42459
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:11:42.639333 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35273
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:42.989850 27778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:42.990170 27778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:42.990314 27778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:42.998083 27778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:42.998241 27778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:42.998432 27778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:11:43.008847 27778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:43.010236 27778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:43.011654 27778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:43.031792 27787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:43.031894 27786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:43.031833 27789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:43.032620 27778 server_base.cc:1034] running on GCE node
I20260812 01:11:43.033277 27778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:43.034274 27778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:43.035492 27778 hybrid_clock.cc:648] HybridClock initialized: now 1786497103035423 us; error 64 us; skew 500 ppm
I20260812 01:11:43.035825 27778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:43 dist-test-slave-21b9 krb5kdc[27357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497103, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:43.039149 27778 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:11:43.040769 27778 webserver.cc:460] Webserver started at http://127.20.151.2:35171/ using document root <none> and password file <none>
I20260812 01:11:43.041497 27778 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:43.041615 27778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:43.041905 27778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:43.045064 27778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "36bad06052b3491c9d12e4b9202113ec"
format_stamp: "Formatted at 2026-08-12 01:11:43 on dist-test-slave-21b9"
server_key: "4afa245a566f171960ba765b11a12dbc"
server_key_iv: "3a4e247e8eb898c0ea52c712987dc68b"
server_key_version: "encryptionkey@0"
I20260812 01:11:43.045859 27778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "36bad06052b3491c9d12e4b9202113ec"
format_stamp: "Formatted at 2026-08-12 01:11:43 on dist-test-slave-21b9"
server_key: "4afa245a566f171960ba765b11a12dbc"
server_key_iv: "3a4e247e8eb898c0ea52c712987dc68b"
server_key_version: "encryptionkey@0"
I20260812 01:11:43.051344 27778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:11:43.055034 27798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:43.056480 27778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 01:11:43.056644 27778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "36bad06052b3491c9d12e4b9202113ec"
format_stamp: "Formatted at 2026-08-12 01:11:43 on dist-test-slave-21b9"
server_key: "4afa245a566f171960ba765b11a12dbc"
server_key_iv: "3a4e247e8eb898c0ea52c712987dc68b"
server_key_version: "encryptionkey@0"
I20260812 01:11:43.056845 27778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:43.071885 27778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:43.076354 27778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:43.076619 27778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:43.077589 27778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:43.078974 27778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:43.079049 27778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:43.079144 27778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:43.079190 27778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:43.119896 27778 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:39225
I20260812 01:11:43.119978 27992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:39225 every 8 connection(s)
I20260812 01:11:43.121186 27778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:11:43.126432 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 27778
I20260812 01:11:43.126642 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:11:43.127187 21084 external_mini_cluster.cc:1428] Setting key 60d00e707c453d334a905c713b8b0796
Aug 12 01:11:43 dist-test-slave-21b9 krb5kdc[27357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497103, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:43.144002 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:43.123869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:45567 (local address 127.20.151.62:35273)
0812 01:11:43.124353 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:43.124358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:43.125299 (+ 941us) server_negotiation.cc:407] Connection header received
0812 01:11:43.127764 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:43.127768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:43.127864 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:43.127995 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:43.129856 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.130582 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.132782 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.133207 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.133301 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:43.137511 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:43.137517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:43.137520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:43.137556 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:43.140542 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.141389 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.141396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.141399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.141562 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.142019 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.142024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.142027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.142389 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:43.142544 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:43.143397 (+ 853us) server_negotiation.cc:299] Negotiation successful
0812 01:11:43.143641 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":236,"threads_started":1}
I20260812 01:11:43.144829 27997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:43.124336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:35273 (local address 127.20.151.2:45567)
0812 01:11:43.125129 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:43.125174 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:43.127431 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:43.128247 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:43.128259 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:43.128862 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:43.129671 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:43.129698 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.130748 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.130754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:43.132654 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:43.132670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.133440 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.133447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:43.133716 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:43.134486 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:43.134515 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:43.137255 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:43.140723 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:43.140732 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:43.140747 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:43.141241 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:43.141683 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:43.141687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:43.141690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:43.141901 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:43.142522 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:43.142528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:43.142875 (+ 347us) client_negotiation.cc:769] Sending connection context
0812 01:11:43.143109 (+ 234us) client_negotiation.cc:240] Negotiation successful
0812 01:11:43.143664 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":281,"threads_started":1}
I20260812 01:11:43.146240 27994 heartbeater.cc:347] Connected to a master server at 127.20.151.62:35273
I20260812 01:11:43.146586 27994 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:43.147593 27994 heartbeater.cc:511] Master 127.20.151.62:35273 requested a full tablet report, sending...
I20260812 01:11:43.149333 27436 ts_manager.cc:194] Registered new tserver with Master: 36bad06052b3491c9d12e4b9202113ec (127.20.151.2:39225)
I20260812 01:11:43.150235 27436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:45567
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:11:43.184703 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35273
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:43.526271 28003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:43.526594 28003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:43.526739 28003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:43.534127 28003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:43.534273 28003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:43.534425 28003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:11:43.544420 28003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:43.545806 28003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:43.547200 28003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:43.560518 28016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:43.560451 28018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:43.560423 28022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:43.561638 28003 server_base.cc:1034] running on GCE node
I20260812 01:11:43.562284 28003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:43.563300 28003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:43.564498 28003 hybrid_clock.cc:648] HybridClock initialized: now 1786497103564439 us; error 53 us; skew 500 ppm
I20260812 01:11:43.564807 28003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:43 dist-test-slave-21b9 krb5kdc[27357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497103, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:43.568039 28003 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:11:43.569578 28003 webserver.cc:460] Webserver started at http://127.20.151.3:44157/ using document root <none> and password file <none>
I20260812 01:11:43.570279 28003 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:43.570396 28003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:43.570719 28003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:43.573907 28003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "ac9900f35fcd458c8106655ec6fc6009"
format_stamp: "Formatted at 2026-08-12 01:11:43 on dist-test-slave-21b9"
server_key: "119424a0ff7f5bceef284a37908687b9"
server_key_iv: "d7b4ffbef429e9c75bd174818cee43bd"
server_key_version: "encryptionkey@0"
I20260812 01:11:43.574718 28003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "ac9900f35fcd458c8106655ec6fc6009"
format_stamp: "Formatted at 2026-08-12 01:11:43 on dist-test-slave-21b9"
server_key: "119424a0ff7f5bceef284a37908687b9"
server_key_iv: "d7b4ffbef429e9c75bd174818cee43bd"
server_key_version: "encryptionkey@0"
I20260812 01:11:43.580698 28003 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260812 01:11:43.584131 28035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:43.585434 28003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:43.585592 28003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "ac9900f35fcd458c8106655ec6fc6009"
format_stamp: "Formatted at 2026-08-12 01:11:43 on dist-test-slave-21b9"
server_key: "119424a0ff7f5bceef284a37908687b9"
server_key_iv: "d7b4ffbef429e9c75bd174818cee43bd"
server_key_version: "encryptionkey@0"
I20260812 01:11:43.585790 28003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:43.608289 27766 heartbeater.cc:503] Master 127.20.151.62:35273 was elected leader, sending a full tablet report...
I20260812 01:11:43.620193 28003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:43.624776 28003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:43.625069 28003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:43.626101 28003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:43.627684 28003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:43.627768 28003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:43.627902 28003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:43.627954 28003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:43.659914 28003 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:40341
I20260812 01:11:43.659963 28214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:40341 every 8 connection(s)
I20260812 01:11:43.661466 28003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:11:43.671103 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 28003
I20260812 01:11:43.671352 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:11:43.671881 21084 external_mini_cluster.cc:1428] Setting key 3bbe0e8ad55571e4c502601dbaacad93
Aug 12 01:11:43 dist-test-slave-21b9 krb5kdc[27357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497103, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:43.682392 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:43.664256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:40571 (local address 127.20.151.62:35273)
0812 01:11:43.664779 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:43.664784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:43.665686 (+ 902us) server_negotiation.cc:407] Connection header received
0812 01:11:43.667608 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:43.667614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:43.667703 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:43.667817 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:43.669383 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.670168 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.672353 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.673012 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.673148 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:43.676795 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:43.676800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:43.676803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:43.676837 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:43.679301 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.680027 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.680031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.680033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.680157 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.680607 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.680612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.680614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.680891 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:43.681029 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:43.681763 (+ 734us) server_negotiation.cc:299] Negotiation successful
0812 01:11:43.682000 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":298,"threads_started":1}
I20260812 01:11:43.683418 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:43.664764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:35273 (local address 127.20.151.3:40571)
0812 01:11:43.665524 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:43.665562 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:43.667323 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:43.668079 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:43.668087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:43.668611 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:43.669215 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:43.669234 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.670310 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.670314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:43.672199 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:43.672214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.673265 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.673274 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:43.673525 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:43.674188 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:43.674218 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:43.676579 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:43.679433 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:43.679443 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:43.679463 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:43.679896 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:43.680262 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:43.680267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:43.680271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:43.680480 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:43.681023 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:43.681031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:43.681382 (+ 351us) client_negotiation.cc:769] Sending connection context
0812 01:11:43.681631 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 01:11:43.682210 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":294,"threads_started":1}
I20260812 01:11:43.684939 28215 heartbeater.cc:347] Connected to a master server at 127.20.151.62:35273
I20260812 01:11:43.685284 28215 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:43.686120 28215 heartbeater.cc:511] Master 127.20.151.62:35273 requested a full tablet report, sending...
I20260812 01:11:43.687803 27436 ts_manager.cc:194] Registered new tserver with Master: ac9900f35fcd458c8106655ec6fc6009 (127.20.151.3:40341)
I20260812 01:11:43.688617 27436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:40571
I20260812 01:11:43.700318 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 01:11:43 dist-test-slave-21b9 krb5kdc[27357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 01:11:43 dist-test-slave-21b9 krb5kdc[27357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:43.761346 28229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:43.745731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.20.151.1:45673)
0812 01:11:43.746423 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:43.746430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:43.746458 (+ 28us) server_negotiation.cc:407] Connection header received
0812 01:11:43.746522 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:43.746529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:43.746752 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:43.746903 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:43.748510 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.749577 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.750821 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.751484 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.751605 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:43.755724 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:43.755739 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:43.755754 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:43.755797 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:43.758512 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.759143 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.759150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.759153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.759323 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.759703 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.759708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.759712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.760086 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:43.760378 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:43.760725 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 01:11:43.761008 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":306,"threads_started":1}
Aug 12 01:11:43 dist-test-slave-21b9 krb5kdc[27357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:43.779949 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:43.766225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.20.151.62:35273)
0812 01:11:43.766441 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:43.766446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:43.766658 (+ 212us) server_negotiation.cc:407] Connection header received
0812 01:11:43.766804 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:43.766808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:43.766887 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:43.766976 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:43.768284 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.769110 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.770388 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.770826 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.770941 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:43.774884 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:43.774889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:43.774892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:43.774927 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:43.777322 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.777972 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.777976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.777978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.778115 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.778485 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.778489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.778492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.778845 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:43.779092 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:43.779409 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 01:11:43.779650 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 01:11:43.780499 27436 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58408
I20260812 01:11:43.781458 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27546
I20260812 01:11:43.794054 28229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:43.782351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.20.151.1:45673)
0812 01:11:43.782518 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:43.782522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:43.782712 (+ 190us) server_negotiation.cc:407] Connection header received
0812 01:11:43.782860 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:43.782864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:43.782955 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:43.783037 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:43.784274 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.785002 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.786192 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.786650 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.786752 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:43.789139 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:43.789157 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:43.789161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:43.789203 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:43.791850 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.792461 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.792466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.792470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.792606 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.792942 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.792946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.792949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.793236 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:43.793436 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:43.793631 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 01:11:43.793807 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 01:11:43.794728 27756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:43.878261 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 27546
I20260812 01:11:43.905867 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27778
Aug 12 01:11:43 dist-test-slave-21b9 krb5kdc[27357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:11:43.922385 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:43.906761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.20.151.2:39225)
0812 01:11:43.907337 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:43.907347 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:43.907375 (+ 28us) server_negotiation.cc:407] Connection header received
0812 01:11:43.907437 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:43.907443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:43.907678 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:43.907829 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:43.909081 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.910142 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.911518 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:43.912060 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:43.912175 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:43.916359 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:43.916374 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:43.916390 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:43.916478 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:43.919489 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.920202 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.920210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.920212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.920349 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:43.920740 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:43.920743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:43.920745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:43.921103 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:43.921350 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:43.921760 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 01:11:43.921999 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":259,"threads_started":1}
I20260812 01:11:43.923942 27982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:44.007442 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 27778
I20260812 01:11:44.036432 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 28003
Aug 12 01:11:44 dist-test-slave-21b9 krb5kdc[27357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:11:44.053496 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:44.037497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.20.151.3:40341)
0812 01:11:44.038182 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:44.038189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:44.038221 (+ 32us) server_negotiation.cc:407] Connection header received
0812 01:11:44.038288 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:44.038296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:44.038567 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:44.038727 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:44.040205 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:44.041262 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:44.042574 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:44.043166 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:44.043309 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:44.047915 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:44.047933 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:44.047951 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:44.047998 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:44.050866 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:44.051633 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:44.051641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:44.051643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:44.051802 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:44.052095 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:44.052098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:44.052100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:44.052470 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:44.052726 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:44.052939 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 01:11:44.053157 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":374,"threads_started":1}
I20260812 01:11:44.054893 28208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:44.138207 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 28003
I20260812 01:11:44.163355 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27379
I20260812 01:11:44.164651 27489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:44.280901 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 27379
2026-08-12T01:11:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4327 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:11:44 dist-test-slave-21b9 krb5kdc[28250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:11:44 dist-test-slave-21b9 krb5kdc[28250](info): set up 2 sockets
Aug 12 01:11:44 dist-test-slave-21b9 krb5kdc[28250](info): commencing operation
krb5kdc: starting...
W20260812 01:11:45.543808 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.005s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:11:45 dist-test-slave-21b9 krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:11:45.806434 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:42237
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:42237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:46.153982 28273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:46.154289 28273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:46.154431 28273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:46.161898 28273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:46.162024 28273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:46.162081 28273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:46.162180 28273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:46.162240 28273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:11:46.172570 28273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:42237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:42237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:46.173913 28273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:46.175257 28273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:46.187165 28278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:46.187098 28280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:46.187135 28287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:46.188165 28273 server_base.cc:1034] running on GCE node
I20260812 01:11:46.189009 28273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:46.190472 28273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:46.191691 28273 hybrid_clock.cc:648] HybridClock initialized: now 1786497106191650 us; error 28 us; skew 500 ppm
I20260812 01:11:46.192041 28273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:46 dist-test-slave-21b9 krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497106, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:46.195246 28273 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:11:46.196507 28273 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:46.196655 28273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:46.196985 28273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:46.200075 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "a433e5b746a34cbb9efae359af1ce4be"
format_stamp: "Formatted at 2026-08-12 01:11:46 on dist-test-slave-21b9"
server_key: "864c7e952e2d00166e41a8214673559c"
server_key_iv: "e9b388f8958e37f29622672613545a11"
server_key_version: "encryptionkey@0"
I20260812 01:11:46.200870 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "a433e5b746a34cbb9efae359af1ce4be"
format_stamp: "Formatted at 2026-08-12 01:11:46 on dist-test-slave-21b9"
server_key: "864c7e952e2d00166e41a8214673559c"
server_key_iv: "e9b388f8958e37f29622672613545a11"
server_key_version: "encryptionkey@0"
I20260812 01:11:46.206089 28273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 01:11:46.209565 28300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:46.210851 28273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 01:11:46.211018 28273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "a433e5b746a34cbb9efae359af1ce4be"
format_stamp: "Formatted at 2026-08-12 01:11:46 on dist-test-slave-21b9"
server_key: "864c7e952e2d00166e41a8214673559c"
server_key_iv: "e9b388f8958e37f29622672613545a11"
server_key_version: "encryptionkey@0"
I20260812 01:11:46.211234 28273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:46.230293 28273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:46.234436 28273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:46.234747 28273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:46.253841 28273 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:42237
I20260812 01:11:46.253865 28390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:42237 every 8 connection(s)
I20260812 01:11:46.254802 28273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:46.258903 28392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:46.262959 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 28273
I20260812 01:11:46.263221 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:46.264094 21084 external_mini_cluster.cc:1428] Setting key ac6654bf04072a3c446b820b6c597fb6
I20260812 01:11:46.272949 28392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be: Bootstrap starting.
I20260812 01:11:46.276477 28392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:46.277607 28392 log.cc:824] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be: Log is configured to *not* fsync() on all Append() calls
Aug 12 01:11:46 dist-test-slave-21b9 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:46.281078 28392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be: No bootstrap required, opened a new log
I20260812 01:11:46.287688 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:46.266438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.20.151.62:42237)
0812 01:11:46.267381 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:46.267395 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:46.267438 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:11:46.269116 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:46.269154 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:46.269642 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:46.270056 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:46.271722 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:46.273078 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:46.274395 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:46.275119 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:46.275338 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:46.279837 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:46.279847 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:46.279864 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:46.279909 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:46.283091 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:46.284055 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:46.284064 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:46.284074 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:46.284262 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:46.284685 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:46.284688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:46.284690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:46.285304 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:46.285540 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:46.286045 (+ 505us) server_negotiation.cc:299] Negotiation successful
0812 01:11:46.286464 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":417,"threads_started":1}
I20260812 01:11:46.290659 28392 raft_consensus.cc:348] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a433e5b746a34cbb9efae359af1ce4be" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 42237 } }
I20260812 01:11:46.291074 28392 raft_consensus.cc:374] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:46.291178 28392 raft_consensus.cc:729] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a433e5b746a34cbb9efae359af1ce4be, State: Initialized, Role: FOLLOWER
I20260812 01:11:46.291757 28392 consensus_queue.cc:260] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a433e5b746a34cbb9efae359af1ce4be" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 42237 } }
I20260812 01:11:46.291993 28392 raft_consensus.cc:388] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:46.292122 28392 raft_consensus.cc:482] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:46.292285 28392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:46.294543 28392 raft_consensus.cc:504] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a433e5b746a34cbb9efae359af1ce4be" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 42237 } }
I20260812 01:11:46.295065 28392 leader_election.cc:302] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a433e5b746a34cbb9efae359af1ce4be; no voters:
I20260812 01:11:46.295712 28392 leader_election.cc:288] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:46.295842 28397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:46.297195 28397 raft_consensus.cc:686] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [term 1 LEADER]: Becoming Leader. State: Replica: a433e5b746a34cbb9efae359af1ce4be, State: Running, Role: LEADER
I20260812 01:11:46.297901 28397 consensus_queue.cc:237] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a433e5b746a34cbb9efae359af1ce4be" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 42237 } }
I20260812 01:11:46.298755 28392 sys_catalog.cc:564] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:46.301028 28399 sys_catalog.cc:455] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [sys.catalog]: SysCatalogTable state changed. Reason: New leader a433e5b746a34cbb9efae359af1ce4be. Latest consensus state: current_term: 1 leader_uuid: "a433e5b746a34cbb9efae359af1ce4be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a433e5b746a34cbb9efae359af1ce4be" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 42237 } } }
I20260812 01:11:46.301501 28399 sys_catalog.cc:458] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:46.301600 28400 sys_catalog.cc:455] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a433e5b746a34cbb9efae359af1ce4be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a433e5b746a34cbb9efae359af1ce4be" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 42237 } } }
I20260812 01:11:46.301946 28400 sys_catalog.cc:458] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:46.303027 28411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:46.308720 28411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:46.318059 28411 catalog_manager.cc:1347] Generated new cluster ID: b4962c5e81d14936a1a4968e681cd688
I20260812 01:11:46.318190 28411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:46.341060 28411 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:46.342525 28411 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:46.360299 28411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be: Generated new TSK 0
I20260812 01:11:46.361421 28411 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:11:46.436789 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:42237
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:46.800964 28433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:46.801272 28433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:46.801373 28433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:46.809002 28433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:46.809132 28433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:46.809283 28433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:11:46.819602 28433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:42237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:46.820859 28433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:46.822146 28433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:46.834685 28439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:46.834758 28441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:46.834703 28438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:46.836200 28433 server_base.cc:1034] running on GCE node
I20260812 01:11:46.836900 28433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:46.837904 28433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:46.839123 28433 hybrid_clock.cc:648] HybridClock initialized: now 1786497106839071 us; error 37 us; skew 500 ppm
I20260812 01:11:46.839546 28433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:46 dist-test-slave-21b9 krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497106, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:46.842782 28433 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:11:46.843986 28433 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:46.844094 28433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:46.844370 28433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:46.847378 28433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "6a5f962665284928a0fac3b44ded688c"
format_stamp: "Formatted at 2026-08-12 01:11:46 on dist-test-slave-21b9"
server_key: "e13b5b7dc9ee575cc9bce21eef69fb3f"
server_key_iv: "3d66cb0f7cd29cc8cc7c7b0394bd42b5"
server_key_version: "encryptionkey@0"
I20260812 01:11:46.848166 28433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "6a5f962665284928a0fac3b44ded688c"
format_stamp: "Formatted at 2026-08-12 01:11:46 on dist-test-slave-21b9"
server_key: "e13b5b7dc9ee575cc9bce21eef69fb3f"
server_key_iv: "3d66cb0f7cd29cc8cc7c7b0394bd42b5"
server_key_version: "encryptionkey@0"
I20260812 01:11:46.853185 28433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 01:11:46.856685 28450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:46.858008 28433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:46.858155 28433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "6a5f962665284928a0fac3b44ded688c"
format_stamp: "Formatted at 2026-08-12 01:11:46 on dist-test-slave-21b9"
server_key: "e13b5b7dc9ee575cc9bce21eef69fb3f"
server_key_iv: "3d66cb0f7cd29cc8cc7c7b0394bd42b5"
server_key_version: "encryptionkey@0"
I20260812 01:11:46.858341 28433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:46.878450 28433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:46.882879 28433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:46.883165 28433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:46.884150 28433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:46.885520 28433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:46.885622 28433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:46.885726 28433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:46.885809 28433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:46.915598 28433 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:34099
I20260812 01:11:46.915668 28641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:34099 every 8 connection(s)
I20260812 01:11:46.916523 28433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:11:46.926563 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 28433
I20260812 01:11:46.926833 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:11:46.927650 21084 external_mini_cluster.cc:1428] Setting key cb117157e3c47d76e396c834c543d115
Aug 12 01:11:46 dist-test-slave-21b9 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497106, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:46.937342 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:46.918950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:42297 (local address 127.20.151.62:42237)
0812 01:11:46.919426 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:46.919430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:46.920459 (+ 1029us) server_negotiation.cc:407] Connection header received
0812 01:11:46.922312 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:46.922316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:46.922386 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:46.922483 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:46.924161 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:46.924869 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:46.926496 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:46.927103 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:46.927246 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:46.931198 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:46.931205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:46.931210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:46.931250 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:46.934200 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:46.934964 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:46.934970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:46.934973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:46.935129 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:46.935572 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:46.935577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:46.935579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:46.935924 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:46.936063 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:46.936763 (+ 700us) server_negotiation.cc:299] Negotiation successful
0812 01:11:46.936987 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":211,"threads_started":1}
I20260812 01:11:46.938210 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:46.919518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:42237 (local address 127.20.151.1:42297)
0812 01:11:46.920291 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:46.920339 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:46.922063 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:46.922773 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:46.922781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:46.923380 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:46.923974 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:46.923997 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:46.925040 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:46.925045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:46.926336 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:46.926350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:46.927369 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:46.927375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:46.927584 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:46.928256 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:46.928285 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:46.930961 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:46.934344 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:46.934352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:46.934365 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:46.934838 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:46.935236 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:46.935240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:46.935242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:46.935464 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:46.936044 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:46.936050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:46.936391 (+ 341us) client_negotiation.cc:769] Sending connection context
0812 01:11:46.936634 (+ 243us) client_negotiation.cc:240] Negotiation successful
0812 01:11:46.937046 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":328,"threads_started":1}
I20260812 01:11:46.939752 28642 heartbeater.cc:347] Connected to a master server at 127.20.151.62:42237
I20260812 01:11:46.940091 28642 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:46.940801 28642 heartbeater.cc:511] Master 127.20.151.62:42237 requested a full tablet report, sending...
I20260812 01:11:46.942931 28328 ts_manager.cc:194] Registered new tserver with Master: 6a5f962665284928a0fac3b44ded688c (127.20.151.1:34099)
I20260812 01:11:46.944443 28328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:42297
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:11:46.982833 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:42237
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:47.334038 28655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:47.334326 28655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:47.334425 28655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:47.341820 28655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:47.341945 28655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:47.342067 28655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:11:47.352056 28655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:42237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:47.353386 28655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:47.354730 28655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:47.367745 28662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:47.367830 28670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:47.367759 28664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:47.368693 28655 server_base.cc:1034] running on GCE node
I20260812 01:11:47.369298 28655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:47.370301 28655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:47.371502 28655 hybrid_clock.cc:648] HybridClock initialized: now 1786497107371446 us; error 45 us; skew 500 ppm
I20260812 01:11:47.371819 28655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:47 dist-test-slave-21b9 krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497107, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:47.374883 28655 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:11:47.376047 28655 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:47.376156 28655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:47.376461 28655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:47.379410 28655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182"
format_stamp: "Formatted at 2026-08-12 01:11:47 on dist-test-slave-21b9"
server_key: "10699826402774209be686e62fc75bc0"
server_key_iv: "ee12ba6635cbd8c55bbe3de8fefeeecb"
server_key_version: "encryptionkey@0"
I20260812 01:11:47.380191 28655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182"
format_stamp: "Formatted at 2026-08-12 01:11:47 on dist-test-slave-21b9"
server_key: "10699826402774209be686e62fc75bc0"
server_key_iv: "ee12ba6635cbd8c55bbe3de8fefeeecb"
server_key_version: "encryptionkey@0"
I20260812 01:11:47.385048 28655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:11:47.388432 28678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:47.389696 28655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:47.389853 28655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182"
format_stamp: "Formatted at 2026-08-12 01:11:47 on dist-test-slave-21b9"
server_key: "10699826402774209be686e62fc75bc0"
server_key_iv: "ee12ba6635cbd8c55bbe3de8fefeeecb"
server_key_version: "encryptionkey@0"
I20260812 01:11:47.390056 28655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:47.407398 28655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:47.411690 28655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:47.411958 28655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:47.412927 28655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:47.414351 28655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:47.414455 28655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:47.414582 28655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:47.414649 28655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:47.446007 28655 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:42041
I20260812 01:11:47.446056 28888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:42041 every 8 connection(s)
I20260812 01:11:47.446843 28655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:11:47.448752 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 28655
I20260812 01:11:47.448958 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:11:47.449559 21084 external_mini_cluster.cc:1428] Setting key 3a43b20c6a0d5e0ab1ccaccc05ed71ea
Aug 12 01:11:47 dist-test-slave-21b9 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497107, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:47.468828 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:47.449347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:35861 (local address 127.20.151.62:42237)
0812 01:11:47.449783 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:47.449788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:47.451015 (+ 1227us) server_negotiation.cc:407] Connection header received
0812 01:11:47.453384 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:47.453388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:47.453455 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:47.453535 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:47.455544 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:47.456250 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:47.458288 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:47.458728 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:47.458818 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:47.462902 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:47.462908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:47.462911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:47.462947 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:47.465571 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:47.466381 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:47.466385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:47.466387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:47.466541 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:47.466987 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:47.466991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:47.466993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:47.467309 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:47.467444 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:47.468207 (+ 763us) server_negotiation.cc:299] Negotiation successful
0812 01:11:47.468445 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":228,"threads_started":1}
I20260812 01:11:47.469883 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:47.450017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:42237 (local address 127.20.151.2:35861)
0812 01:11:47.450858 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:47.450904 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:47.453076 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:47.453805 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:47.453816 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:47.454561 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:47.455370 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:47.455401 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:47.456407 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:47.456412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:47.458172 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:47.458186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:47.458950 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:47.458957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:47.459236 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:47.459978 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:47.460006 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:47.462640 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:47.465729 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:47.465740 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:47.465759 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:47.466247 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:47.466655 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:47.466660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:47.466664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:47.466869 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:47.467436 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:47.467444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:47.467810 (+ 366us) client_negotiation.cc:769] Sending connection context
0812 01:11:47.468069 (+ 259us) client_negotiation.cc:240] Negotiation successful
0812 01:11:47.468661 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":328,"threads_started":1}
I20260812 01:11:47.471442 28890 heartbeater.cc:347] Connected to a master server at 127.20.151.62:42237
I20260812 01:11:47.471828 28890 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:47.472746 28890 heartbeater.cc:511] Master 127.20.151.62:42237 requested a full tablet report, sending...
I20260812 01:11:47.474411 28328 ts_manager.cc:194] Registered new tserver with Master: bcd26cc8f97a48cf8e3bffb4bc1d6182 (127.20.151.2:42041)
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
I20260812 01:11:47.475327 28328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:35861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:11:47.507378 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:42237
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:47.847662 28900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:47.847968 28900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:47.848078 28900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:47.855432 28900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:47.855562 28900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:47.855690 28900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:11:47.865733 28900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:42237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:47.867017 28900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:47.868363 28900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:47.881079 28913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:47.881134 28914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:47.881160 28916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:47.882129 28900 server_base.cc:1034] running on GCE node
I20260812 01:11:47.882697 28900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:47.883641 28900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:47.884850 28900 hybrid_clock.cc:648] HybridClock initialized: now 1786497107884801 us; error 39 us; skew 500 ppm
I20260812 01:11:47.885172 28900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:47 dist-test-slave-21b9 krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497107, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:47.888226 28900 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:11:47.889366 28900 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:47.889457 28900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:47.889725 28900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:47.892789 28900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "2cd26fbb1e554e998db90cba1f5b581b"
format_stamp: "Formatted at 2026-08-12 01:11:47 on dist-test-slave-21b9"
server_key: "baf0c5c4215d3b7730b7d0e6b7364907"
server_key_iv: "c413eecb1633a6dc660d593e4246a552"
server_key_version: "encryptionkey@0"
I20260812 01:11:47.893631 28900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "2cd26fbb1e554e998db90cba1f5b581b"
format_stamp: "Formatted at 2026-08-12 01:11:47 on dist-test-slave-21b9"
server_key: "baf0c5c4215d3b7730b7d0e6b7364907"
server_key_iv: "c413eecb1633a6dc660d593e4246a552"
server_key_version: "encryptionkey@0"
I20260812 01:11:47.898828 28900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260812 01:11:47.902498 28927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:47.903977 28900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 01:11:47.904134 28900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "2cd26fbb1e554e998db90cba1f5b581b"
format_stamp: "Formatted at 2026-08-12 01:11:47 on dist-test-slave-21b9"
server_key: "baf0c5c4215d3b7730b7d0e6b7364907"
server_key_iv: "c413eecb1633a6dc660d593e4246a552"
server_key_version: "encryptionkey@0"
I20260812 01:11:47.904346 28900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:47.930536 28900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:47.934978 28900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:47.935343 28900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:47.936414 28900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:47.937866 28900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:47.937956 28900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:47.938071 28900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:47.938139 28900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:47.947454 28642 heartbeater.cc:503] Master 127.20.151.62:42237 was elected leader, sending a full tablet report...
I20260812 01:11:47.970206 28900 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:42547
I20260812 01:11:47.970260 29127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:42547 every 8 connection(s)
I20260812 01:11:47.971071 28900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:11:47.973345 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 28900
I20260812 01:11:47.973588 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:11:47.974128 21084 external_mini_cluster.cc:1428] Setting key 90daefee0b77115d1a9dfacc9d1c632d
Aug 12 01:11:47 dist-test-slave-21b9 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497107, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:47.990665 29133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:47.973543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:33313 (local address 127.20.151.62:42237)
0812 01:11:47.974062 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:47.974068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:47.975084 (+ 1016us) server_negotiation.cc:407] Connection header received
0812 01:11:47.977010 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:47.977016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:47.977101 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:47.977196 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:47.978715 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:47.979531 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:47.981042 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:47.981465 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:47.981584 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:47.985155 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:47.985161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:47.985164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:47.985197 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:47.987656 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:47.988374 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:47.988379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:47.988381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:47.988510 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:47.988914 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:47.988918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:47.988920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:47.989221 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:47.989363 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:47.990053 (+ 690us) server_negotiation.cc:299] Negotiation successful
0812 01:11:47.990276 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":240,"threads_started":1}
I20260812 01:11:47.991663 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:47.974135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:42237 (local address 127.20.151.3:33313)
0812 01:11:47.974931 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:47.974996 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:47.976752 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:47.977443 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:47.977451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:47.977963 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:47.978541 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:47.978559 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:47.979681 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:47.979685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:47.980900 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:47.980912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:47.981702 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:47.981709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:47.981914 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:47.982630 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:47.982653 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:47.984923 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:47.987804 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:47.987813 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:47.987830 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:47.988263 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:47.988615 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:47.988620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:47.988624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:47.988806 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:47.989348 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:47.989356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:47.989697 (+ 341us) client_negotiation.cc:769] Sending connection context
0812 01:11:47.989949 (+ 252us) client_negotiation.cc:240] Negotiation successful
0812 01:11:47.990494 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":325,"threads_started":1}
I20260812 01:11:47.993085 29128 heartbeater.cc:347] Connected to a master server at 127.20.151.62:42237
I20260812 01:11:47.993391 29128 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:47.994096 29128 heartbeater.cc:511] Master 127.20.151.62:42237 requested a full tablet report, sending...
I20260812 01:11:47.995536 28328 ts_manager.cc:194] Registered new tserver with Master: 2cd26fbb1e554e998db90cba1f5b581b (127.20.151.3:42547)
I20260812 01:11:47.996330 28328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:33313
I20260812 01:11:48.001343 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:11:48.020659 29133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.006098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.20.151.62:42237)
0812 01:11:48.006282 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:48.006287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:48.006751 (+ 464us) server_negotiation.cc:407] Connection header received
0812 01:11:48.006879 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:48.006883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:48.006958 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:48.007049 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:48.008717 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.009869 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.011814 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.012441 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.012573 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.015634 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:48.015639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:48.015642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:48.015680 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:48.018335 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:48.018936 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:48.018940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:48.018942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:48.019069 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:48.019398 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:48.019401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:48.019403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:48.019788 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:48.019983 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:48.020232 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 01:11:48.020408 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:11:48.024828 28328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:11:48.031950 28328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:11:48.067689 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.045073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.20.151.1:34099)
0812 01:11:48.055996 (+ 10923us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:48.056007 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:48.056911 (+ 904us) server_negotiation.cc:407] Connection header received
0812 01:11:48.057049 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:48.057057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:48.057325 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:48.057543 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:48.058576 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.060390 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.064465 (+ 4075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.066070 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.066218 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.067057 (+ 839us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:48.067134 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 01:11:48.067329 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10685,"thread_start_us":10944,"threads_started":1}
I20260812 01:11:48.068665 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.044711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34099 (local address 127.0.0.1:40122)
0812 01:11:48.056648 (+ 11937us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.056666 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:48.056784 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:48.057883 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:48.057892 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:48.057929 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:48.058350 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:48.058364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.060633 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.060640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:48.064292 (+ 3652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:48.064309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.066289 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.066295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:48.066530 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.066615 (+ 85us) client_negotiation.cc:769] Sending connection context
0812 01:11:48.066746 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 01:11:48.068168 (+ 1422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11792,"mutex_wait_us":1145,"thread_start_us":11136,"threads_started":1}
I20260812 01:11:48.071851 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.044860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.20.151.2:42041)
0812 01:11:48.062868 (+ 18008us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:48.062878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:48.062915 (+ 37us) server_negotiation.cc:407] Connection header received
0812 01:11:48.063030 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:48.063038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:48.064393 (+ 1355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:48.064606 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:48.065247 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.066233 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.069026 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.070345 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.070484 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.070702 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:48.071380 (+ 678us) server_negotiation.cc:299] Negotiation successful
0812 01:11:48.071559 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17794,"thread_start_us":17657,"threads_started":1}
I20260812 01:11:48.072283 29156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.057237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:42547 (local address 127.0.0.1:44434)
0812 01:11:48.063426 (+ 6189us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.063450 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:48.063598 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:48.064847 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:48.064856 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:48.064882 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:48.065340 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:48.065351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.067445 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.067450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:48.069533 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:48.069545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.071434 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.071441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:48.071636 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.071668 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:11:48.071735 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 01:11:48.071858 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5955,"mutex_wait_us":26,"thread_start_us":5780,"threads_started":1}
I20260812 01:11:48.072402 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.044696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:42041 (local address 127.0.0.1:48994)
0812 01:11:48.056295 (+ 11599us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.056351 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:48.056594 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:48.064613 (+ 8019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:48.064617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:48.064642 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:48.065086 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:48.065096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.066389 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.066394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:48.068850 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:48.068866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.070849 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.070856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:48.071033 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.071065 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:11:48.071149 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 01:11:48.072052 (+ 903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11286,"mutex_wait_us":2,"thread_start_us":12063,"threads_started":1}
I20260812 01:11:48.073331 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.057715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44434 (local address 127.20.151.3:42547)
0812 01:11:48.064214 (+ 6499us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:48.064221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:48.064255 (+ 34us) server_negotiation.cc:407] Connection header received
0812 01:11:48.064349 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:48.064357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:48.064625 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:48.065536 (+ 911us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:48.065848 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.067316 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.069698 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.071162 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.072544 (+ 1382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.072769 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:48.072840 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 01:11:48.073024 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6337,"thread_start_us":6146,"threads_started":1}
I20260812 01:11:48.073486 28534 tablet_service.cc:1467] Processing CreateTablet for tablet 918890b1463e477882b802a3c704fdb6 (DEFAULT_TABLE table=test-table [id=e1e1972ac909469f8b568f2d3bddbed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:48.075853 28534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918890b1463e477882b802a3c704fdb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:48.076313 28777 tablet_service.cc:1467] Processing CreateTablet for tablet 918890b1463e477882b802a3c704fdb6 (DEFAULT_TABLE table=test-table [id=e1e1972ac909469f8b568f2d3bddbed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:48.077483 28777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918890b1463e477882b802a3c704fdb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:48.077224 29010 tablet_service.cc:1467] Processing CreateTablet for tablet 918890b1463e477882b802a3c704fdb6 (DEFAULT_TABLE table=test-table [id=e1e1972ac909469f8b568f2d3bddbed4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:48.078606 29010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918890b1463e477882b802a3c704fdb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:48.089108 29160 tablet_bootstrap.cc:492] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: Bootstrap starting.
I20260812 01:11:48.092423 29160 tablet_bootstrap.cc:654] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:48.093389 29162 tablet_bootstrap.cc:492] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: Bootstrap starting.
I20260812 01:11:48.094345 29160 log.cc:824] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:48.096870 29161 tablet_bootstrap.cc:492] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c: Bootstrap starting.
I20260812 01:11:48.098444 29162 tablet_bootstrap.cc:654] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:48.099781 29162 log.cc:824] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:48.100337 29161 tablet_bootstrap.cc:654] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:48.100948 29160 tablet_bootstrap.cc:492] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: No bootstrap required, opened a new log
I20260812 01:11:48.101219 29160 ts_tablet_manager.cc:1397] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260812 01:11:48.101580 29161 log.cc:824] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:48.103039 29162 tablet_bootstrap.cc:492] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: No bootstrap required, opened a new log
I20260812 01:11:48.103448 29162 ts_tablet_manager.cc:1397] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260812 01:11:48.104817 29161 tablet_bootstrap.cc:492] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c: No bootstrap required, opened a new log
I20260812 01:11:48.105072 29161 ts_tablet_manager.cc:1397] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260812 01:11:48.112090 29161 raft_consensus.cc:348] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42041 } } peers { permanent_uuid: "6a5f962665284928a0fac3b44ded688c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34099 } } peers { permanent_uuid: "2cd26fbb1e554e998db90cba1f5b581b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42547 } }
I20260812 01:11:48.111949 29160 raft_consensus.cc:348] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42041 } } peers { permanent_uuid: "6a5f962665284928a0fac3b44ded688c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34099 } } peers { permanent_uuid: "2cd26fbb1e554e998db90cba1f5b581b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42547 } }
I20260812 01:11:48.112435 29160 raft_consensus.cc:374] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:48.112440 29161 raft_consensus.cc:374] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:48.112537 29161 raft_consensus.cc:729] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a5f962665284928a0fac3b44ded688c, State: Initialized, Role: FOLLOWER
I20260812 01:11:48.112567 29160 raft_consensus.cc:729] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd26cc8f97a48cf8e3bffb4bc1d6182, State: Initialized, Role: FOLLOWER
I20260812 01:11:48.113126 29161 consensus_queue.cc:260] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42041 } } peers { permanent_uuid: "6a5f962665284928a0fac3b44ded688c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34099 } } peers { permanent_uuid: "2cd26fbb1e554e998db90cba1f5b581b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42547 } }
I20260812 01:11:48.113294 29160 consensus_queue.cc:260] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42041 } } peers { permanent_uuid: "6a5f962665284928a0fac3b44ded688c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34099 } } peers { permanent_uuid: "2cd26fbb1e554e998db90cba1f5b581b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42547 } }
I20260812 01:11:48.114082 29162 raft_consensus.cc:348] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42041 } } peers { permanent_uuid: "6a5f962665284928a0fac3b44ded688c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34099 } } peers { permanent_uuid: "2cd26fbb1e554e998db90cba1f5b581b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42547 } }
I20260812 01:11:48.114589 29162 raft_consensus.cc:374] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:48.114728 29162 raft_consensus.cc:729] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd26fbb1e554e998db90cba1f5b581b, State: Initialized, Role: FOLLOWER
I20260812 01:11:48.115382 29162 consensus_queue.cc:260] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42041 } } peers { permanent_uuid: "6a5f962665284928a0fac3b44ded688c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34099 } } peers { permanent_uuid: "2cd26fbb1e554e998db90cba1f5b581b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42547 } }
I20260812 01:11:48.117344 29161 ts_tablet_manager.cc:1428] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c: Time spent starting tablet: real 0.012s user 0.006s sys 0.003s
I20260812 01:11:48.121928 28890 heartbeater.cc:503] Master 127.20.151.62:42237 was elected leader, sending a full tablet report...
I20260812 01:11:48.123035 29160 ts_tablet_manager.cc:1428] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260812 01:11:48.123996 29128 heartbeater.cc:503] Master 127.20.151.62:42237 was elected leader, sending a full tablet report...
I20260812 01:11:48.124922 29162 ts_tablet_manager.cc:1428] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: Time spent starting tablet: real 0.021s user 0.016s sys 0.001s
I20260812 01:11:48.144625 29174 raft_consensus.cc:482] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:11:48.144964 29174 raft_consensus.cc:504] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42041 } } peers { permanent_uuid: "6a5f962665284928a0fac3b44ded688c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34099 } } peers { permanent_uuid: "2cd26fbb1e554e998db90cba1f5b581b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42547 } }
I20260812 01:11:48.146889 29174 leader_election.cc:288] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a5f962665284928a0fac3b44ded688c (127.20.151.1:34099), 2cd26fbb1e554e998db90cba1f5b581b (127.20.151.3:42547)
I20260812 01:11:48.153473 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.147989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:38509 (local address 127.20.151.1:34099)
0812 01:11:48.148207 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:48.148213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:48.148418 (+ 205us) server_negotiation.cc:407] Connection header received
0812 01:11:48.148577 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:48.148581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:48.148645 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:48.148717 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:48.149303 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.150019 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.151815 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.152550 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.152713 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.152787 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:48.153096 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 01:11:48.153224 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 01:11:48.153489 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.147508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:39419 (local address 127.20.151.3:42547)
0812 01:11:48.147726 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:48.147733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:48.148093 (+ 360us) server_negotiation.cc:407] Connection header received
0812 01:11:48.148411 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:48.148416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:48.148478 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:48.148555 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:11:48.149125 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.149885 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.151773 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.152548 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.152705 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.152787 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:48.153120 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 01:11:48.153249 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 01:11:48.153637 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.147240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34099 (local address 127.20.151.2:38509)
0812 01:11:48.148313 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.148328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:48.148454 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:48.148760 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:48.148765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:48.148794 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:48.149163 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:48.149172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.150157 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.150161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:48.151672 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:48.151685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.152779 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.152785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:48.152950 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.152975 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:11:48.153056 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 01:11:48.153200 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":22,"thread_start_us":730,"threads_started":1}
I20260812 01:11:48.153936 29180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.147154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:42547 (local address 127.20.151.2:39419)
0812 01:11:48.147972 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.147994 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:48.148240 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:48.148583 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:48.148588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:48.148614 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:11:48.148977 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:48.148989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.150011 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.150015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:48.151628 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:48.151641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.152817 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.152824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:48.152974 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.152998 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 01:11:48.153078 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 01:11:48.153203 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":52,"thread_start_us":367,"threads_started":1}
I20260812 01:11:48.154881 28565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918890b1463e477882b802a3c704fdb6" candidate_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a5f962665284928a0fac3b44ded688c" is_pre_election: true
I20260812 01:11:48.154881 29049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918890b1463e477882b802a3c704fdb6" candidate_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd26fbb1e554e998db90cba1f5b581b" is_pre_election: true
I20260812 01:11:48.155540 29049 raft_consensus.cc:2454] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd26cc8f97a48cf8e3bffb4bc1d6182 in term 0.
I20260812 01:11:48.155540 28565 raft_consensus.cc:2454] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd26cc8f97a48cf8e3bffb4bc1d6182 in term 0.
I20260812 01:11:48.156222 28682 leader_election.cc:302] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd26fbb1e554e998db90cba1f5b581b, bcd26cc8f97a48cf8e3bffb4bc1d6182; no voters:
I20260812 01:11:48.156635 29174 raft_consensus.cc:2781] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:11:48.156775 29174 raft_consensus.cc:482] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:11:48.156877 29174 raft_consensus.cc:3037] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:48.159080 29174 raft_consensus.cc:504] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42041 } } peers { permanent_uuid: "6a5f962665284928a0fac3b44ded688c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34099 } } peers { permanent_uuid: "2cd26fbb1e554e998db90cba1f5b581b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42547 } }
I20260812 01:11:48.159708 29174 leader_election.cc:288] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [CANDIDATE]: Term 1 election: Requested vote from peers 6a5f962665284928a0fac3b44ded688c (127.20.151.1:34099), 2cd26fbb1e554e998db90cba1f5b581b (127.20.151.3:42547)
I20260812 01:11:48.160127 28565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918890b1463e477882b802a3c704fdb6" candidate_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a5f962665284928a0fac3b44ded688c"
I20260812 01:11:48.160153 29049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918890b1463e477882b802a3c704fdb6" candidate_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd26fbb1e554e998db90cba1f5b581b"
I20260812 01:11:48.160310 28565 raft_consensus.cc:3037] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:48.160322 29049 raft_consensus.cc:3037] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:48.162545 29049 raft_consensus.cc:2454] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd26cc8f97a48cf8e3bffb4bc1d6182 in term 1.
I20260812 01:11:48.162580 28565 raft_consensus.cc:2454] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd26cc8f97a48cf8e3bffb4bc1d6182 in term 1.
I20260812 01:11:48.162945 28682 leader_election.cc:302] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd26fbb1e554e998db90cba1f5b581b, bcd26cc8f97a48cf8e3bffb4bc1d6182; no voters:
I20260812 01:11:48.163290 29174 raft_consensus.cc:2781] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:48.163724 29174 raft_consensus.cc:686] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 1 LEADER]: Becoming Leader. State: Replica: bcd26cc8f97a48cf8e3bffb4bc1d6182, State: Running, Role: LEADER
I20260812 01:11:48.164251 29174 consensus_queue.cc:237] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42041 } } peers { permanent_uuid: "6a5f962665284928a0fac3b44ded688c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34099 } } peers { permanent_uuid: "2cd26fbb1e554e998db90cba1f5b581b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42547 } }
W20260812 01:11:48.168689 28643 tablet.cc:2369] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:48.169296 28326 catalog_manager.cc:5495] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcd26cc8f97a48cf8e3bffb4bc1d6182 (127.20.151.2). New cstate: current_term: 1 leader_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd26cc8f97a48cf8e3bffb4bc1d6182" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a5f962665284928a0fac3b44ded688c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd26fbb1e554e998db90cba1f5b581b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42547 } health_report { overall_health: UNKNOWN } } }
W20260812 01:11:48.198612 28891 tablet.cc:2369] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:48.201798 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.196622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.20.151.2:42041)
0812 01:11:48.196822 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:48.196827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:48.196845 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:11:48.196987 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:48.196990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:48.197071 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:48.197200 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:11:48.197504 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.198219 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.199748 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.200218 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.200314 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.200731 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:11:48.201225 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:11:48.201331 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:48.201446 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 01:11:48.201515 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:11:48.210038 28565 raft_consensus.cc:1261] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c [term 1 FOLLOWER]: Refusing update from remote peer bcd26cc8f97a48cf8e3bffb4bc1d6182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:48.210035 29049 raft_consensus.cc:1261] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b [term 1 FOLLOWER]: Refusing update from remote peer bcd26cc8f97a48cf8e3bffb4bc1d6182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:48.211035 29183 consensus_queue.cc:1035] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a5f962665284928a0fac3b44ded688c" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:48.211433 29174 consensus_queue.cc:1035] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd26fbb1e554e998db90cba1f5b581b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 01:11:48.222477 29132 tablet.cc:2369] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:48.223693 29190 mvcc.cc:205] Tried to move back new op lower bound from 7317492155218595840 to 7317492155045560320. Current Snapshot: MvccSnapshot[applied={T|T < 7317492155218595840}]
I20260812 01:11:48.237694 28328 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:48.238027 28328 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:48.242043 28328 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a433e5b746a34cbb9efae359af1ce4be: Sending DeleteTablet for 3 replicas of tablet 918890b1463e477882b802a3c704fdb6
I20260812 01:11:48.243211 28777 tablet_service.cc:1514] Processing DeleteTablet for tablet 918890b1463e477882b802a3c704fdb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:48 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:48994
I20260812 01:11:48.243257 28534 tablet_service.cc:1514] Processing DeleteTablet for tablet 918890b1463e477882b802a3c704fdb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:48 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:40122
I20260812 01:11:48.243551 29010 tablet_service.cc:1514] Processing DeleteTablet for tablet 918890b1463e477882b802a3c704fdb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:48 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:44434
I20260812 01:11:48.245500 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 28433
I20260812 01:11:48.269966 29208 tablet_replica.cc:330] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: stopping tablet replica
I20260812 01:11:48.273959 29211 tablet_replica.cc:330] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: stopping tablet replica
I20260812 01:11:48.277900 29211 raft_consensus.cc:2229] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:48.278733 29211 raft_consensus.cc:2258] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:48.279779 29208 raft_consensus.cc:2229] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:11:48.280634 29208 raft_consensus.cc:2258] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:48.281944 29206 tablet_replica.cc:330] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c: stopping tablet replica
I20260812 01:11:48.283155 29211 ts_tablet_manager.cc:1905] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:48.283737 29208 ts_tablet_manager.cc:1905] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:48.284256 29206 raft_consensus.cc:2229] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:48.286392 29206 raft_consensus.cc:2258] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 12 01:11:48 dist-test-slave-21b9 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:48.293040 29208 ts_tablet_manager.cc:1918] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:11:48.293345 29208 log.cc:1196] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/918890b1463e477882b802a3c704fdb6
I20260812 01:11:48.294148 29211 ts_tablet_manager.cc:1918] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:11:48.294209 29208 ts_tablet_manager.cc:1939] T 918890b1463e477882b802a3c704fdb6 P bcd26cc8f97a48cf8e3bffb4bc1d6182: Deleting consensus metadata
I20260812 01:11:48.294440 29211 log.cc:1196] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/918890b1463e477882b802a3c704fdb6
I20260812 01:11:48.294631 29206 ts_tablet_manager.cc:1905] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:48.295145 29211 ts_tablet_manager.cc:1939] T 918890b1463e477882b802a3c704fdb6 P 2cd26fbb1e554e998db90cba1f5b581b: Deleting consensus metadata
I20260812 01:11:48.296126 28306 catalog_manager.cc:4841] TS bcd26cc8f97a48cf8e3bffb4bc1d6182 (127.20.151.2:42041): tablet 918890b1463e477882b802a3c704fdb6 (table test-table [id=e1e1972ac909469f8b568f2d3bddbed4]) successfully deleted
I20260812 01:11:48.297014 28306 catalog_manager.cc:4841] TS 2cd26fbb1e554e998db90cba1f5b581b (127.20.151.3:42547): tablet 918890b1463e477882b802a3c704fdb6 (table test-table [id=e1e1972ac909469f8b568f2d3bddbed4]) successfully deleted
I20260812 01:11:48.299191 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.247192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.20.151.1:34099)
0812 01:11:48.247412 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:48.247418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:48.270722 (+ 23304us) server_negotiation.cc:407] Connection header received
0812 01:11:48.271365 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:48.271372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:48.271487 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:48.271572 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:48.274275 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.275546 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.279294 (+ 3748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.280005 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.280128 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.287974 (+ 7846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:48.287989 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:48.288006 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:48.288058 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:48.291741 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:48.293514 (+ 1773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:48.293526 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:48.293530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:48.293796 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:48.296941 (+ 3145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:48.296947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:48.296951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:48.297471 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:48.297785 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:48.298473 (+ 688us) server_negotiation.cc:299] Negotiation successful
0812 01:11:48.298745 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 01:11:48.300241 28636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:48.391064 29206 ts_tablet_manager.cc:1918] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:11:48.391395 29206 log.cc:1196] T 918890b1463e477882b802a3c704fdb6 P 6a5f962665284928a0fac3b44ded688c: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/918890b1463e477882b802a3c704fdb6
I20260812 01:11:48.391812 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 28433
W20260812 01:11:48.425191 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.151.1:34099 (error 108)
I20260812 01:11:48.425460 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 28655
I20260812 01:11:48.427672 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.427168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34099 (local address 127.0.0.1:40152)
0812 01:11:48.427361 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.427489 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 01:11:48.428781 28307 catalog_manager.cc:4584] TS 6a5f962665284928a0fac3b44ded688c (127.20.151.1:34099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 918890b1463e477882b802a3c704fdb6: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34099: connect: Connection refused (error 111)
Aug 12 01:11:48 dist-test-slave-21b9 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:11:48.442693 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.426663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49020 (local address 127.20.151.2:42041)
0812 01:11:48.426895 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:48.426904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:48.426924 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:11:48.427105 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:48.427111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:48.427200 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:48.427333 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:48.428717 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.429971 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.431195 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.431894 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.432025 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.436186 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:48.436203 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:48.436227 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:48.436284 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:48.440020 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:48.440640 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:48.440650 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:48.440653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:48.440831 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:48.441145 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:48.441149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:48.441152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:48.441589 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:48.441912 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:48.442086 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 01:11:48.442310 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 01:11:48.443401 28883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:48.482353 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.481786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34099 (local address 127.0.0.1:40160)
0812 01:11:48.482032 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.482148 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 01:11:48.483433 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.483024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34099 (local address 127.0.0.1:40164)
0812 01:11:48.483161 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.483248 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 01:11:48.544180 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.543616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34099 (local address 127.0.0.1:40170)
0812 01:11:48.543859 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.543974 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 01:11:48.545215 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.544848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34099 (local address 127.0.0.1:40180)
0812 01:11:48.544988 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.545071 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 01:11:48.561558 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 28655
I20260812 01:11:48.589650 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 28900
Aug 12 01:11:48 dist-test-slave-21b9 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:11:48.605240 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.590420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.20.151.3:42547)
0812 01:11:48.590632 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:48.590637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:48.590656 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:11:48.590803 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:48.590807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:48.590884 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:48.590968 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:48.592257 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.592998 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.594220 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:48.594653 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:48.594755 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:48.599899 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:48.599916 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:48.599934 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:48.599984 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:48.602566 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:48.603337 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:48.603345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:48.603347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:48.603499 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:48.603814 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:48.603816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:48.603818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:48.604220 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:48.604461 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:48.604789 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 01:11:48.604966 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 01:11:48.605914 29122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:48.625924 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.625339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34099 (local address 127.0.0.1:40190)
0812 01:11:48.625611 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.625734 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 01:11:48.627081 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.626619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34099 (local address 127.0.0.1:40198)
0812 01:11:48.626841 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.626932 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 01:11:48.695894 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 28900
I20260812 01:11:48.717242 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 28273
I20260812 01:11:48.718372 28381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:48.830132 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:48.829423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34099 (local address 127.0.0.1:40210)
0812 01:11:48.829759 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:48.829880 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260812 01:11:48.830437 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 28273
2026-08-12T01:11:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4551 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T01:11:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:48Z Disabled control of system clock
I20260812 01:11:48.920572 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:37085
--webserver_interface=127.20.151.62
--webserver_port=0
--builtin_ntp_servers=127.20.151.20:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:37085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 01:11:49.265142 29232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:49.265451 29232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:49.265583 29232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:49.273051 29232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:49.273182 29232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:49.273224 29232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:49.273306 29232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:49.273367 29232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:11:49.282948 29232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:37085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:37085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:49.284332 29232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:49.286289 29232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:49.298403 29237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:49.298372 29241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:49.298455 29239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:49.299374 29232 server_base.cc:1034] running on GCE node
I20260812 01:11:49.300194 29232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:49.301666 29232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:49.302906 29232 hybrid_clock.cc:648] HybridClock initialized: now 1786497109302853 us; error 42 us; skew 500 ppm
I20260812 01:11:49.303318 29232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:11:49.305727 29232 webserver.cc:460] Webserver started at http://127.20.151.62:34599/ using document root <none> and password file <none>
I20260812 01:11:49.306479 29232 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:49.306576 29232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:49.306916 29232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:49.309966 29232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "57622a0860a8495fa3bfa3d2ee249161"
format_stamp: "Formatted at 2026-08-12 01:11:49 on dist-test-slave-21b9"
server_key: "1638901648cf7ecb7bcfbee16fca6d0b"
server_key_iv: "e92370ed03270d09f9d812ec08e4c50d"
server_key_version: "encryptionkey@0"
I20260812 01:11:49.310806 29232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "57622a0860a8495fa3bfa3d2ee249161"
format_stamp: "Formatted at 2026-08-12 01:11:49 on dist-test-slave-21b9"
server_key: "1638901648cf7ecb7bcfbee16fca6d0b"
server_key_iv: "e92370ed03270d09f9d812ec08e4c50d"
server_key_version: "encryptionkey@0"
I20260812 01:11:49.316166 29232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260812 01:11:49.319841 29249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:49.321180 29232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 01:11:49.321349 29232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "57622a0860a8495fa3bfa3d2ee249161"
format_stamp: "Formatted at 2026-08-12 01:11:49 on dist-test-slave-21b9"
server_key: "1638901648cf7ecb7bcfbee16fca6d0b"
server_key_iv: "e92370ed03270d09f9d812ec08e4c50d"
server_key_version: "encryptionkey@0"
I20260812 01:11:49.321578 29232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:49.327553 29232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:49.328366 29232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:49.328657 29232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:49.346696 29232 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:37085
I20260812 01:11:49.346737 29342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:37085 every 8 connection(s)
I20260812 01:11:49.348079 29232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:49.352119 29343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:49.356891 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 29232
I20260812 01:11:49.357105 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:49.357957 21084 external_mini_cluster.cc:1428] Setting key 3c12ba3c62e554e151e594cb45e04721
I20260812 01:11:49.364293 29343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161: Bootstrap starting.
I20260812 01:11:49.366864 29346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:49.360206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.20.151.62:37085)
0812 01:11:49.361284 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:49.361296 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:49.361341 (+ 45us) server_negotiation.cc:407] Connection header received
0812 01:11:49.363177 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:49.363203 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:49.363678 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:49.364050 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:49.365160 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:49.365166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:49.365180 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:49.365203 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:49.365266 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:49.365283 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:49.365524 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:49.365675 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 01:11:49.365943 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":381,"threads_started":1}
I20260812 01:11:49.367764 29343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:49.368847 29343 log.cc:824] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:49.371694 29343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161: No bootstrap required, opened a new log
I20260812 01:11:49.379048 29343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57622a0860a8495fa3bfa3d2ee249161" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37085 } }
I20260812 01:11:49.379498 29343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:49.379603 29343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57622a0860a8495fa3bfa3d2ee249161, State: Initialized, Role: FOLLOWER
I20260812 01:11:49.380189 29343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57622a0860a8495fa3bfa3d2ee249161" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37085 } }
I20260812 01:11:49.380417 29343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:49.380554 29343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:49.380709 29343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:49.382944 29343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57622a0860a8495fa3bfa3d2ee249161" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37085 } }
I20260812 01:11:49.383580 29343 leader_election.cc:302] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57622a0860a8495fa3bfa3d2ee249161; no voters:
I20260812 01:11:49.384166 29343 leader_election.cc:288] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:49.384306 29351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:49.385298 29351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [term 1 LEADER]: Becoming Leader. State: Replica: 57622a0860a8495fa3bfa3d2ee249161, State: Running, Role: LEADER
I20260812 01:11:49.385861 29351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57622a0860a8495fa3bfa3d2ee249161" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37085 } }
I20260812 01:11:49.386415 29343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:49.388764 29353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57622a0860a8495fa3bfa3d2ee249161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57622a0860a8495fa3bfa3d2ee249161" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37085 } } }
I20260812 01:11:49.389187 29353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:49.389681 29354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57622a0860a8495fa3bfa3d2ee249161. Latest consensus state: current_term: 1 leader_uuid: "57622a0860a8495fa3bfa3d2ee249161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57622a0860a8495fa3bfa3d2ee249161" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37085 } } }
I20260812 01:11:49.390015 29354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:49.391126 29365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:49.396852 29365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:49.406287 29365 catalog_manager.cc:1347] Generated new cluster ID: 97e7039234c847d9ac26c8e014275d68
I20260812 01:11:49.406435 29365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:49.432140 29365 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:49.432629 29365 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:49.441958 29365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161: Generated new TSK 0
I20260812 01:11:49.442812 29365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:11:49.455363 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:37085
--builtin_ntp_servers=127.20.151.20:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 01:11:49.807416 29380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:49.807758 29380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:49.807910 29380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:49.815569 29380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:49.815694 29380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:49.815852 29380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:11:49.825429 29380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:49.826813 29380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:49.828677 29380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:49.841972 29387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:49.842022 29388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:49.842079 29391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:49.843195 29380 server_base.cc:1034] running on GCE node
I20260812 01:11:49.843896 29380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:49.844898 29380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:49.846099 29380 hybrid_clock.cc:648] HybridClock initialized: now 1786497109846046 us; error 45 us; skew 500 ppm
I20260812 01:11:49.846424 29380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:11:49.848886 29380 webserver.cc:460] Webserver started at http://127.20.151.1:44611/ using document root <none> and password file <none>
I20260812 01:11:49.849640 29380 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:49.849745 29380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:49.850083 29380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:49.853183 29380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "2c9c798016d2449e8a2b6160004142df"
format_stamp: "Formatted at 2026-08-12 01:11:49 on dist-test-slave-21b9"
server_key: "2124602412ed4977d836baa10ae4b99d"
server_key_iv: "e3557eaabab4d967aee032ee9228fdf3"
server_key_version: "encryptionkey@0"
I20260812 01:11:49.853984 29380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "2c9c798016d2449e8a2b6160004142df"
format_stamp: "Formatted at 2026-08-12 01:11:49 on dist-test-slave-21b9"
server_key: "2124602412ed4977d836baa10ae4b99d"
server_key_iv: "e3557eaabab4d967aee032ee9228fdf3"
server_key_version: "encryptionkey@0"
I20260812 01:11:49.859182 29380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:11:49.862692 29401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:49.864076 29380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 01:11:49.864238 29380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "2c9c798016d2449e8a2b6160004142df"
format_stamp: "Formatted at 2026-08-12 01:11:49 on dist-test-slave-21b9"
server_key: "2124602412ed4977d836baa10ae4b99d"
server_key_iv: "e3557eaabab4d967aee032ee9228fdf3"
server_key_version: "encryptionkey@0"
I20260812 01:11:49.864426 29380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:49.869845 29380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:49.870615 29380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:49.870842 29380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:49.871729 29380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:49.873072 29380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:49.873158 29380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:49.873240 29380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:49.873294 29380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:49.904618 29380 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:43027
I20260812 01:11:49.904670 29588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:43027 every 8 connection(s)
I20260812 01:11:49.905998 29380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:11:49.913355 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 29380
I20260812 01:11:49.913533 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:11:49.914353 21084 external_mini_cluster.cc:1428] Setting key 0b0e4a0e38c7635df21c908b20ce93b7
I20260812 01:11:49.914584 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:49.908717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:43717 (local address 127.20.151.62:37085)
0812 01:11:49.909217 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:49.909224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:49.910493 (+ 1269us) server_negotiation.cc:407] Connection header received
0812 01:11:49.912287 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:49.912291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:49.912347 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:49.912446 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:49.913834 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:49.913840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:49.913845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:49.913868 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:49.913925 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:49.913941 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:49.914076 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:49.914207 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 01:11:49.914275 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":224,"threads_started":1}
I20260812 01:11:49.915652 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:49.909362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:37085 (local address 127.20.151.1:43717)
0812 01:11:49.910334 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:49.910372 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:49.912037 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:49.912690 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:49.912698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:49.912793 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:49.913503 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:11:49.913507 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:49.913551 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:11:49.913661 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:49.914029 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:49.914035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:49.914040 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 01:11:49.914257 (+ 217us) client_negotiation.cc:240] Negotiation successful
0812 01:11:49.914537 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":290,"threads_started":1}
I20260812 01:11:49.917191 29591 heartbeater.cc:347] Connected to a master server at 127.20.151.62:37085
I20260812 01:11:49.917507 29591 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:49.917585 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:37085
--builtin_ntp_servers=127.20.151.20:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 01:11:49.918147 29591 heartbeater.cc:511] Master 127.20.151.62:37085 requested a full tablet report, sending...
I20260812 01:11:49.920496 29281 ts_manager.cc:194] Registered new tserver with Master: 2c9c798016d2449e8a2b6160004142df (127.20.151.1:43027)
W20260812 01:11:50.271893 29596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:50.272205 29596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:50.272344 29596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:50.279786 29596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:50.279898 29596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:50.280067 29596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:11:50.289682 29596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:50.291072 29596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:50.292954 29596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:50.306674 29609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:50.306748 29614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:50.306777 29608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:50.307651 29596 server_base.cc:1034] running on GCE node
I20260812 01:11:50.308286 29596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:50.309279 29596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:50.310478 29596 hybrid_clock.cc:648] HybridClock initialized: now 1786497110310431 us; error 35 us; skew 500 ppm
I20260812 01:11:50.310812 29596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:11:50.313177 29596 webserver.cc:460] Webserver started at http://127.20.151.2:43783/ using document root <none> and password file <none>
I20260812 01:11:50.313897 29596 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:50.313992 29596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:50.314352 29596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:50.317371 29596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce"
format_stamp: "Formatted at 2026-08-12 01:11:50 on dist-test-slave-21b9"
server_key: "a036db53527ce38d2f5fbe55687cb226"
server_key_iv: "a8ec5c677e7b8854c43ef0f573df379a"
server_key_version: "encryptionkey@0"
I20260812 01:11:50.318194 29596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce"
format_stamp: "Formatted at 2026-08-12 01:11:50 on dist-test-slave-21b9"
server_key: "a036db53527ce38d2f5fbe55687cb226"
server_key_iv: "a8ec5c677e7b8854c43ef0f573df379a"
server_key_version: "encryptionkey@0"
I20260812 01:11:50.323487 29596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260812 01:11:50.327111 29624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:50.328590 29596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:50.328755 29596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce"
format_stamp: "Formatted at 2026-08-12 01:11:50 on dist-test-slave-21b9"
server_key: "a036db53527ce38d2f5fbe55687cb226"
server_key_iv: "a8ec5c677e7b8854c43ef0f573df379a"
server_key_version: "encryptionkey@0"
I20260812 01:11:50.328984 29596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:50.334836 29596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:50.335736 29596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:50.335997 29596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:50.336967 29596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:50.338419 29596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:50.338510 29596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:50.338621 29596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:50.338693 29596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:50.378257 29596 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:37189
I20260812 01:11:50.378311 29829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:37189 every 8 connection(s)
I20260812 01:11:50.379647 29596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:11:50.386344 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 29596
I20260812 01:11:50.386524 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:11:50.386992 21084 external_mini_cluster.cc:1428] Setting key 8a1cf1797856c9a70575947f4256980c
I20260812 01:11:50.388362 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:50.382236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:42153 (local address 127.20.151.62:37085)
0812 01:11:50.382440 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:50.382445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:50.383925 (+ 1480us) server_negotiation.cc:407] Connection header received
0812 01:11:50.385760 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:50.385766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:50.385859 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:50.385971 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:50.387590 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:50.387597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:50.387602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:50.387626 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:50.387674 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:50.387693 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:50.387824 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:50.388049 (+ 225us) server_negotiation.cc:299] Negotiation successful
0812 01:11:50.388110 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 01:11:50.389494 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:50.382715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:37085 (local address 127.20.151.2:42153)
0812 01:11:50.383744 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:50.383783 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:50.385460 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:50.386251 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:50.386263 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:50.386369 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:50.387193 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:11:50.387196 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:50.387249 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:11:50.387403 (+ 154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:50.387815 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:50.387825 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:50.387832 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 01:11:50.388129 (+ 297us) client_negotiation.cc:240] Negotiation successful
0812 01:11:50.388390 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":276,"threads_started":1}
I20260812 01:11:50.390188 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:37085
--builtin_ntp_servers=127.20.151.20:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 01:11:50.390813 29831 heartbeater.cc:347] Connected to a master server at 127.20.151.62:37085
I20260812 01:11:50.391094 29831 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:50.391741 29831 heartbeater.cc:511] Master 127.20.151.62:37085 requested a full tablet report, sending...
I20260812 01:11:50.393186 29281 ts_manager.cc:194] Registered new tserver with Master: 0534a9cf71ec4139af2ee9ca05f2f0ce (127.20.151.2:37189)
W20260812 01:11:50.744962 29835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:50.745275 29835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:50.745432 29835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:50.752988 29835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:50.753116 29835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:50.753278 29835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:11:50.762684 29835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:37085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:50.764111 29835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:50.765944 29835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:50.779513 29847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:50.779543 29841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:50.779791 29840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:50.780543 29835 server_base.cc:1034] running on GCE node
I20260812 01:11:50.781148 29835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:50.782166 29835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:50.783380 29835 hybrid_clock.cc:648] HybridClock initialized: now 1786497110783315 us; error 51 us; skew 500 ppm
I20260812 01:11:50.783722 29835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:11:50.786127 29835 webserver.cc:460] Webserver started at http://127.20.151.3:34337/ using document root <none> and password file <none>
I20260812 01:11:50.786913 29835 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:50.787019 29835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:50.787425 29835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:50.790889 29835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "aefd474ca4364c518bd3107ba01b9e83"
format_stamp: "Formatted at 2026-08-12 01:11:50 on dist-test-slave-21b9"
server_key: "ab6f3476b5ff9dd49a4a74f8fe936bed"
server_key_iv: "d75aabeb1bd733e2d9b6c45fa9c88f13"
server_key_version: "encryptionkey@0"
I20260812 01:11:50.791730 29835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "aefd474ca4364c518bd3107ba01b9e83"
format_stamp: "Formatted at 2026-08-12 01:11:50 on dist-test-slave-21b9"
server_key: "ab6f3476b5ff9dd49a4a74f8fe936bed"
server_key_iv: "d75aabeb1bd733e2d9b6c45fa9c88f13"
server_key_version: "encryptionkey@0"
I20260812 01:11:50.797098 29835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:11:50.800760 29860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:50.802122 29835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:50.802290 29835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "aefd474ca4364c518bd3107ba01b9e83"
format_stamp: "Formatted at 2026-08-12 01:11:50 on dist-test-slave-21b9"
server_key: "ab6f3476b5ff9dd49a4a74f8fe936bed"
server_key_iv: "d75aabeb1bd733e2d9b6c45fa9c88f13"
server_key_version: "encryptionkey@0"
I20260812 01:11:50.802515 29835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:50.808542 29835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:50.809381 29835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:50.809650 29835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:50.810619 29835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:50.812112 29835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:50.812227 29835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:50.812338 29835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:50.812412 29835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:50.851928 29835 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:37259
I20260812 01:11:50.852013 30056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:37259 every 8 connection(s)
I20260812 01:11:50.853263 29835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:11:50.856993 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 29835
I20260812 01:11:50.857182 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:11:50.857741 21084 external_mini_cluster.cc:1428] Setting key 81451e5c9fd5b7feb0605ed2d4b941c7
I20260812 01:11:50.862334 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:50.856213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:35885 (local address 127.20.151.62:37085)
0812 01:11:50.856430 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:50.856437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:50.857941 (+ 1504us) server_negotiation.cc:407] Connection header received
0812 01:11:50.860013 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:50.860018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:50.860110 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:50.860197 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:50.861546 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:50.861554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:50.861559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:50.861581 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:50.861627 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:50.861643 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:50.861766 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:50.861965 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 01:11:50.862034 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 01:11:50.863129 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:50.856727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:37085 (local address 127.20.151.3:35885)
0812 01:11:50.857777 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:50.857821 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:50.859738 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:50.860437 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:50.860446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:50.860535 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:50.861232 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:11:50.861235 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:50.861274 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:11:50.861356 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:50.861761 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:50.861770 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:50.861777 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 01:11:50.861997 (+ 220us) client_negotiation.cc:240] Negotiation successful
0812 01:11:50.862190 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":310,"threads_started":1}
I20260812 01:11:50.864444 30057 heartbeater.cc:347] Connected to a master server at 127.20.151.62:37085
I20260812 01:11:50.864763 30057 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:50.865464 30057 heartbeater.cc:511] Master 127.20.151.62:37085 requested a full tablet report, sending...
I20260812 01:11:50.867022 29281 ts_manager.cc:194] Registered new tserver with Master: aefd474ca4364c518bd3107ba01b9e83 (127.20.151.3:37259)
I20260812 01:11:50.879230 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:11:50.886737 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:50.884066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.20.151.62:37085)
0812 01:11:50.884285 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:50.884293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:50.884775 (+ 482us) server_negotiation.cc:407] Connection header received
0812 01:11:50.884967 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:50.884971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:50.885029 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:50.885109 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:50.886090 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:50.886094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:50.886097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:50.886113 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:50.886147 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:50.886158 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:50.886253 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:50.886344 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 01:11:50.886402 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:11:50.891141 29281 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:11:50.898383 29281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:11:50.923867 29591 heartbeater.cc:503] Master 127.20.151.62:37085 was elected leader, sending a full tablet report...
I20260812 01:11:50.925413 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:50.912265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:37259 (local address 127.0.0.1:47264)
0812 01:11:50.913088 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:50.913128 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:50.913260 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:50.920923 (+ 7663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:50.920932 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:50.921066 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:50.922124 (+ 1058us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:11:50.922129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:50.922168 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:11:50.922273 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:50.924627 (+ 2354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:50.924643 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:50.924652 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 01:11:50.924846 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 01:11:50.924951 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"threads_started":1}
I20260812 01:11:50.925951 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:50.913136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.20.151.3:37259)
0812 01:11:50.920215 (+ 7079us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:50.920225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:50.920271 (+ 46us) server_negotiation.cc:407] Connection header received
0812 01:11:50.920355 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:50.920364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:50.920640 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:50.920832 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:50.924370 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:50.924381 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:50.924399 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:50.924438 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:50.924503 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:50.924525 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:50.924711 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:50.925251 (+ 540us) server_negotiation.cc:299] Negotiation successful
0812 01:11:50.925543 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6876,"thread_start_us":6684,"threads_started":1}
I20260812 01:11:50.935498 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:50.912380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43027 (local address 127.0.0.1:56854)
0812 01:11:50.925634 (+ 13254us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:50.925652 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:50.925729 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:50.932613 (+ 6884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:50.932619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:50.932653 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:50.933414 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:11:50.933418 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:50.933454 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:11:50.933520 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:50.934941 (+ 1421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:50.934947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:50.934954 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 01:11:50.935063 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 01:11:50.935127 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13153,"thread_start_us":269,"threads_started":1}
I20260812 01:11:50.935612 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:50.913686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.20.151.1:43027)
0812 01:11:50.931830 (+ 18144us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:50.931911 (+ 81us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:50.931957 (+ 46us) server_negotiation.cc:407] Connection header received
0812 01:11:50.932047 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:50.932057 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:50.932342 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:50.932539 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:50.934692 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:50.934701 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:50.934716 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:50.934733 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:50.934803 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:50.934825 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:50.935024 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:50.935092 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 01:11:50.935199 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17961,"thread_start_us":17800,"threads_started":1}
I20260812 01:11:50.938023 29947 tablet_service.cc:1467] Processing CreateTablet for tablet 8455f99f3996436ba02280f1adb81869 (DEFAULT_TABLE table=test-table [id=b671e726e7b5480ebe3ae331a6f4e9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:50.939224 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:50.913664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:37189 (local address 127.0.0.1:40236)
0812 01:11:50.933545 (+ 19881us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:50.933564 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:50.933677 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:50.937261 (+ 3584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:50.937266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:50.937291 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:50.938061 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:11:50.938064 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:50.938082 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:11:50.938131 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:50.938625 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:50.938630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:50.938635 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 01:11:50.938767 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 01:11:50.938824 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19725,"thread_start_us":24101,"threads_started":1}
I20260812 01:11:50.939771 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:50.913971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.20.151.2:37189)
0812 01:11:50.928590 (+ 14619us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:50.928600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:50.933793 (+ 5193us) server_negotiation.cc:407] Connection header received
0812 01:11:50.935964 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:50.935975 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:50.936204 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:50.936340 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:50.938315 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:50.938325 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:50.938338 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:50.938367 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:50.938425 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:50.938446 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:50.938566 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:50.939346 (+ 780us) server_negotiation.cc:299] Negotiation successful
0812 01:11:50.939448 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14426,"thread_start_us":14625,"threads_started":1}
I20260812 01:11:50.939821 29947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8455f99f3996436ba02280f1adb81869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:50.943197 29490 tablet_service.cc:1467] Processing CreateTablet for tablet 8455f99f3996436ba02280f1adb81869 (DEFAULT_TABLE table=test-table [id=b671e726e7b5480ebe3ae331a6f4e9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:50.943255 29716 tablet_service.cc:1467] Processing CreateTablet for tablet 8455f99f3996436ba02280f1adb81869 (DEFAULT_TABLE table=test-table [id=b671e726e7b5480ebe3ae331a6f4e9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:11:50.944633 29490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8455f99f3996436ba02280f1adb81869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:50.944643 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8455f99f3996436ba02280f1adb81869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:50.955087 30090 tablet_bootstrap.cc:492] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: Bootstrap starting.
I20260812 01:11:50.958873 30090 tablet_bootstrap.cc:654] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:50.960186 30090 log.cc:824] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:50.966190 30093 tablet_bootstrap.cc:492] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df: Bootstrap starting.
I20260812 01:11:50.967031 30090 tablet_bootstrap.cc:492] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: No bootstrap required, opened a new log
I20260812 01:11:50.967340 30090 ts_tablet_manager.cc:1397] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260812 01:11:50.967588 30095 tablet_bootstrap.cc:492] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: Bootstrap starting.
I20260812 01:11:50.970078 30093 tablet_bootstrap.cc:654] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:50.971309 30095 tablet_bootstrap.cc:654] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:50.971422 30093 log.cc:824] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:50.972455 30095 log.cc:824] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:50.974574 30093 tablet_bootstrap.cc:492] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df: No bootstrap required, opened a new log
I20260812 01:11:50.974915 30093 ts_tablet_manager.cc:1397] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260812 01:11:50.976084 30095 tablet_bootstrap.cc:492] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: No bootstrap required, opened a new log
I20260812 01:11:50.976353 30095 ts_tablet_manager.cc:1397] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260812 01:11:50.976418 30090 raft_consensus.cc:348] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefd474ca4364c518bd3107ba01b9e83" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37259 } } peers { permanent_uuid: "2c9c798016d2449e8a2b6160004142df" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43027 } } peers { permanent_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 37189 } }
I20260812 01:11:50.976805 30090 raft_consensus.cc:374] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:50.976907 30090 raft_consensus.cc:729] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefd474ca4364c518bd3107ba01b9e83, State: Initialized, Role: FOLLOWER
I20260812 01:11:50.977473 30090 consensus_queue.cc:260] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefd474ca4364c518bd3107ba01b9e83" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37259 } } peers { permanent_uuid: "2c9c798016d2449e8a2b6160004142df" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43027 } } peers { permanent_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 37189 } }
I20260812 01:11:50.978557 30057 heartbeater.cc:503] Master 127.20.151.62:37085 was elected leader, sending a full tablet report...
I20260812 01:11:50.979027 30090 ts_tablet_manager.cc:1428] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260812 01:11:50.984102 30093 raft_consensus.cc:348] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefd474ca4364c518bd3107ba01b9e83" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37259 } } peers { permanent_uuid: "2c9c798016d2449e8a2b6160004142df" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43027 } } peers { permanent_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 37189 } }
I20260812 01:11:50.984625 30093 raft_consensus.cc:374] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:50.984732 30093 raft_consensus.cc:729] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c9c798016d2449e8a2b6160004142df, State: Initialized, Role: FOLLOWER
I20260812 01:11:50.985489 30093 consensus_queue.cc:260] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefd474ca4364c518bd3107ba01b9e83" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37259 } } peers { permanent_uuid: "2c9c798016d2449e8a2b6160004142df" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43027 } } peers { permanent_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 37189 } }
I20260812 01:11:50.986830 30095 raft_consensus.cc:348] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefd474ca4364c518bd3107ba01b9e83" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37259 } } peers { permanent_uuid: "2c9c798016d2449e8a2b6160004142df" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43027 } } peers { permanent_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 37189 } }
I20260812 01:11:50.987243 30095 raft_consensus.cc:374] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:50.987375 30095 raft_consensus.cc:729] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0534a9cf71ec4139af2ee9ca05f2f0ce, State: Initialized, Role: FOLLOWER
I20260812 01:11:50.988034 30095 consensus_queue.cc:260] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefd474ca4364c518bd3107ba01b9e83" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37259 } } peers { permanent_uuid: "2c9c798016d2449e8a2b6160004142df" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43027 } } peers { permanent_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 37189 } }
I20260812 01:11:50.988526 30093 ts_tablet_manager.cc:1428] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260812 01:11:50.989187 29831 heartbeater.cc:503] Master 127.20.151.62:37085 was elected leader, sending a full tablet report...
I20260812 01:11:50.989764 30095 ts_tablet_manager.cc:1428] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
W20260812 01:11:51.105041 30062 tablet.cc:2369] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:51.108987 30100 raft_consensus.cc:482] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:11:51.109302 30100 raft_consensus.cc:504] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefd474ca4364c518bd3107ba01b9e83" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37259 } } peers { permanent_uuid: "2c9c798016d2449e8a2b6160004142df" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43027 } } peers { permanent_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 37189 } }
I20260812 01:11:51.111017 30100 leader_election.cc:288] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c9c798016d2449e8a2b6160004142df (127.20.151.1:43027), 0534a9cf71ec4139af2ee9ca05f2f0ce (127.20.151.2:37189)
I20260812 01:11:51.113583 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.111570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:50629 (local address 127.20.151.1:43027)
0812 01:11:51.111759 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:51.111766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:51.111827 (+ 61us) server_negotiation.cc:407] Connection header received
0812 01:11:51.111899 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:51.111904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:51.111970 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:51.112073 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:51.112998 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:51.113004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:51.113008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:51.113024 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:51.113057 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:51.113069 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:51.113178 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:51.113275 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 01:11:51.113335 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:11:51.113582 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.111366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43027 (local address 127.20.151.3:50629)
0812 01:11:51.111612 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:51.111633 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:51.111743 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:51.112136 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:51.112142 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:51.112167 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:51.112764 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:11:51.112766 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:51.112782 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:11:51.112831 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:51.113158 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:51.113163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:51.113169 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 01:11:51.113270 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 01:11:51.113330 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 01:11:51.114240 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.111877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:56797 (local address 127.20.151.2:37189)
0812 01:11:51.112078 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:51.112084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:51.112410 (+ 326us) server_negotiation.cc:407] Connection header received
0812 01:11:51.112542 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:51.112546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:51.112601 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:51.112681 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:51.113538 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:51.113543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:51.113546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:51.113561 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:51.113599 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:51.113611 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:51.113710 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:51.113938 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 01:11:51.114000 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 01:11:51.114333 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.111480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:37189 (local address 127.20.151.3:56797)
0812 01:11:51.112299 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:51.112316 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:51.112429 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:51.112726 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:51.112730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:51.112752 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:51.113348 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:11:51.113350 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:51.113363 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:11:51.113399 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:51.113767 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:51.113771 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:51.113775 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 01:11:51.113883 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 01:11:51.113931 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":605,"threads_started":1}
I20260812 01:11:51.114723 29518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8455f99f3996436ba02280f1adb81869" candidate_uuid: "aefd474ca4364c518bd3107ba01b9e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c9c798016d2449e8a2b6160004142df" is_pre_election: true
I20260812 01:11:51.115365 29518 raft_consensus.cc:2454] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefd474ca4364c518bd3107ba01b9e83 in term 0.
I20260812 01:11:51.115340 29747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8455f99f3996436ba02280f1adb81869" candidate_uuid: "aefd474ca4364c518bd3107ba01b9e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" is_pre_election: true
I20260812 01:11:51.115890 29747 raft_consensus.cc:2454] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefd474ca4364c518bd3107ba01b9e83 in term 0.
I20260812 01:11:51.116123 29862 leader_election.cc:302] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c9c798016d2449e8a2b6160004142df, aefd474ca4364c518bd3107ba01b9e83; no voters:
I20260812 01:11:51.116515 30100 raft_consensus.cc:2781] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:11:51.116663 30100 raft_consensus.cc:482] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:11:51.116770 30100 raft_consensus.cc:3037] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:51.118955 30100 raft_consensus.cc:504] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefd474ca4364c518bd3107ba01b9e83" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37259 } } peers { permanent_uuid: "2c9c798016d2449e8a2b6160004142df" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43027 } } peers { permanent_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 37189 } }
I20260812 01:11:51.119597 30100 leader_election.cc:288] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [CANDIDATE]: Term 1 election: Requested vote from peers 2c9c798016d2449e8a2b6160004142df (127.20.151.1:43027), 0534a9cf71ec4139af2ee9ca05f2f0ce (127.20.151.2:37189)
I20260812 01:11:51.120035 29518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8455f99f3996436ba02280f1adb81869" candidate_uuid: "aefd474ca4364c518bd3107ba01b9e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c9c798016d2449e8a2b6160004142df"
I20260812 01:11:51.120072 29747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8455f99f3996436ba02280f1adb81869" candidate_uuid: "aefd474ca4364c518bd3107ba01b9e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce"
I20260812 01:11:51.120230 29518 raft_consensus.cc:3037] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:51.120255 29747 raft_consensus.cc:3037] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:51.122437 29518 raft_consensus.cc:2454] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefd474ca4364c518bd3107ba01b9e83 in term 1.
I20260812 01:11:51.122527 29747 raft_consensus.cc:2454] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefd474ca4364c518bd3107ba01b9e83 in term 1.
I20260812 01:11:51.122879 29862 leader_election.cc:302] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c9c798016d2449e8a2b6160004142df, aefd474ca4364c518bd3107ba01b9e83; no voters:
I20260812 01:11:51.123189 30100 raft_consensus.cc:2781] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:51.123649 30100 raft_consensus.cc:686] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 1 LEADER]: Becoming Leader. State: Replica: aefd474ca4364c518bd3107ba01b9e83, State: Running, Role: LEADER
I20260812 01:11:51.124092 30100 consensus_queue.cc:237] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefd474ca4364c518bd3107ba01b9e83" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37259 } } peers { permanent_uuid: "2c9c798016d2449e8a2b6160004142df" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43027 } } peers { permanent_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 37189 } }
I20260812 01:11:51.128926 29281 catalog_manager.cc:5495] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefd474ca4364c518bd3107ba01b9e83 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "aefd474ca4364c518bd3107ba01b9e83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefd474ca4364c518bd3107ba01b9e83" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 37259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c9c798016d2449e8a2b6160004142df" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 37189 } health_report { overall_health: UNKNOWN } } }
W20260812 01:11:51.131587 29833 tablet.cc:2369] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:51.145465 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.143332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.20.151.3:37259)
0812 01:11:51.143516 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:51.143523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:51.143652 (+ 129us) server_negotiation.cc:407] Connection header received
0812 01:11:51.143866 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:51.143870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:51.143951 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:51.144035 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:51.144890 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:51.144896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:51.144900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:51.144917 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:51.144950 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:51.144962 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:51.145066 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:51.145157 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 01:11:51.145220 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 01:11:51.153899 29747 raft_consensus.cc:1261] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce [term 1 FOLLOWER]: Refusing update from remote peer aefd474ca4364c518bd3107ba01b9e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:51.153903 29518 raft_consensus.cc:1261] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df [term 1 FOLLOWER]: Refusing update from remote peer aefd474ca4364c518bd3107ba01b9e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:11:51.154950 30109 consensus_queue.cc:1035] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0534a9cf71ec4139af2ee9ca05f2f0ce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:11:51.155346 30100 consensus_queue.cc:1035] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c9c798016d2449e8a2b6160004142df" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 01:11:51.158458 29592 tablet.cc:2369] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:11:51.169636 30123 mvcc.cc:205] Tried to move back new op lower bound from 7317492167275134976 to 7317492167168151552. Current Snapshot: MvccSnapshot[applied={T|T < 7317492167275134976}]
I20260812 01:11:51.181116 29279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:51.181425 29279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:11:51.185302 29279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57622a0860a8495fa3bfa3d2ee249161: Sending DeleteTablet for 3 replicas of tablet 8455f99f3996436ba02280f1adb81869
I20260812 01:11:51.186674 29947 tablet_service.cc:1514] Processing DeleteTablet for tablet 8455f99f3996436ba02280f1adb81869 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:51 UTC) from {username='slave'} at 127.0.0.1:47264
I20260812 01:11:51.186684 29490 tablet_service.cc:1514] Processing DeleteTablet for tablet 8455f99f3996436ba02280f1adb81869 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:51 UTC) from {username='slave'} at 127.0.0.1:56854
I20260812 01:11:51.187129 29716 tablet_service.cc:1514] Processing DeleteTablet for tablet 8455f99f3996436ba02280f1adb81869 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:11:51 UTC) from {username='slave'} at 127.0.0.1:40236
I20260812 01:11:51.188606 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 29380
I20260812 01:11:51.191424 30140 tablet_replica.cc:330] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: stopping tablet replica
I20260812 01:11:51.191954 30140 raft_consensus.cc:2229] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:11:51.192675 30140 raft_consensus.cc:2258] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:51.193219 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.189843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.20.151.1:43027)
0812 01:11:51.190026 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:51.190032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:51.190741 (+ 709us) server_negotiation.cc:407] Connection header received
0812 01:11:51.190916 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:51.190921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:51.191042 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:51.191137 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:51.192365 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:51.192371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:51.192376 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:51.192396 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:51.192437 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:51.192453 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:51.192567 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:51.192835 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 01:11:51.192900 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:11:51.194275 29579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:51.195333 30140 ts_tablet_manager.cc:1905] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:51.200268 30135 tablet_replica.cc:330] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: stopping tablet replica
I20260812 01:11:51.200755 30135 raft_consensus.cc:2229] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:11:51.201303 30140 ts_tablet_manager.cc:1918] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:11:51.201472 30140 log.cc:1196] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/8455f99f3996436ba02280f1adb81869
I20260812 01:11:51.201610 30135 raft_consensus.cc:2258] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:11:51.201999 30140 ts_tablet_manager.cc:1939] T 8455f99f3996436ba02280f1adb81869 P 0534a9cf71ec4139af2ee9ca05f2f0ce: Deleting consensus metadata
I20260812 01:11:51.203882 30135 ts_tablet_manager.cc:1905] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:11:51.203917 29259 catalog_manager.cc:4841] TS 0534a9cf71ec4139af2ee9ca05f2f0ce (127.20.151.2:37189): tablet 8455f99f3996436ba02280f1adb81869 (table test-table [id=b671e726e7b5480ebe3ae331a6f4e9d9]) successfully deleted
I20260812 01:11:51.211519 30135 ts_tablet_manager.cc:1918] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:11:51.211727 30135 log.cc:1196] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/8455f99f3996436ba02280f1adb81869
I20260812 01:11:51.212296 30135 ts_tablet_manager.cc:1939] T 8455f99f3996436ba02280f1adb81869 P aefd474ca4364c518bd3107ba01b9e83: Deleting consensus metadata
I20260812 01:11:51.214340 29253 catalog_manager.cc:4841] TS aefd474ca4364c518bd3107ba01b9e83 (127.20.151.3:37259): tablet 8455f99f3996436ba02280f1adb81869 (table test-table [id=b671e726e7b5480ebe3ae331a6f4e9d9]) successfully deleted
I20260812 01:11:51.287894 30137 tablet_replica.cc:330] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df: stopping tablet replica
I20260812 01:11:51.288327 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 29380
I20260812 01:11:51.288412 30137 raft_consensus.cc:2229] T 8455f99f3996436ba02280f1adb81869 P 2c9c798016d2449e8a2b6160004142df [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 01:11:51.313701 29257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.151.1:43027 (error 108)
I20260812 01:11:51.314100 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 29596
I20260812 01:11:51.316077 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.315513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43027 (local address 127.0.0.1:56874)
0812 01:11:51.315728 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:51.315886 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:43027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 01:11:51.316372 29257 catalog_manager.cc:4584] TS 2c9c798016d2449e8a2b6160004142df (127.20.151.1:43027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8455f99f3996436ba02280f1adb81869: Network error: Client connection negotiation failed: client connection to 127.20.151.1:43027: connect: Connection refused (error 111)
I20260812 01:11:51.317425 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.315041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.20.151.2:37189)
0812 01:11:51.315224 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:51.315228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:51.315247 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:11:51.315474 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:51.315480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:51.315611 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:51.315713 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:51.316779 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:51.316785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:51.316790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:51.316808 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:51.316842 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:51.316854 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:51.316965 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:51.317085 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 01:11:51.317148 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 01:11:51.318150 29821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:51.370400 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.369764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43027 (local address 127.0.0.1:56882)
0812 01:11:51.370070 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:51.370182 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:43027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260812 01:11:51.371632 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.371182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43027 (local address 127.0.0.1:56892)
0812 01:11:51.371397 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:51.371495 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:43027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 01:11:51.410115 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 29596
I20260812 01:11:51.432472 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.431866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43027 (local address 127.0.0.1:56894)
0812 01:11:51.432122 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:51.432258 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:43027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 01:11:51.433823 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.433384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43027 (local address 127.0.0.1:56906)
0812 01:11:51.433551 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:51.433663 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:43027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 01:11:51.438812 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 29835
I20260812 01:11:51.442022 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.439886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.20.151.3:37259)
0812 01:11:51.440053 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:51.440058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:51.440077 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:11:51.440312 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:51.440317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:51.440368 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:51.440447 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:51.441378 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:51.441382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:51.441386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:51.441401 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:51.441435 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:51.441447 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:51.441551 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:51.441667 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 01:11:51.441717 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 01:11:51.442764 30048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:51.515102 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.514544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43027 (local address 127.0.0.1:56916)
0812 01:11:51.514800 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:51.514910 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:43027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 01:11:51.516407 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.515914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43027 (local address 127.0.0.1:56918)
0812 01:11:51.516178 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:51.516271 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:43027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260812 01:11:51.566798 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 29835
I20260812 01:11:51.593647 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 29232
I20260812 01:11:51.594907 29336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:51.705770 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:51.704878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43027 (local address 127.0.0.1:56934)
0812 01:11:51.705426 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:51.705550 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:43027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359}
I20260812 01:11:51.705880 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 29232
2026-08-12T01:11:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2874 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 01:11:51.759202 21084 mini_oidc.cc:152] Starting JWKS server
I20260812 01:11:51.762189 21084 webserver.cc:460] Webserver started at https://127.0.0.1:42377/ using document root <none> and password file <none>
I20260812 01:11:51.762593 21084 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 01:11:51.763361 21084 webserver.cc:460] Webserver started at http://127.0.0.1:37745/ using document root <none> and password file <none>
2026-08-12T01:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:51Z Disabled control of system clock
I20260812 01:11:51.802775 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:38487
--webserver_interface=127.20.151.62
--webserver_port=0
--builtin_ntp_servers=127.20.151.20:42781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:38487
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42377/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/testchainca.cert with env {}
W20260812 01:11:52.145045 30164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:52.145349 30164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:52.145465 30164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:52.152799 30164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:52.152921 30164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:52.152961 30164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:52.153039 30164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:52.153100 30164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 01:11:52.153199 30164 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 01:11:52.153250 30164 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42377/jwks.json
I20260812 01:11:52.163538 30164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:42781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:38487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:38487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42377/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:52.164925 30164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:52.166266 30164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:52.177923 30172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:52.177955 30173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:52.178174 30176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:52.178843 30164 server_base.cc:1034] running on GCE node
I20260812 01:11:52.179661 30164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:52.181020 30164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:52.182224 30164 hybrid_clock.cc:648] HybridClock initialized: now 1786497112182178 us; error 32 us; skew 500 ppm
I20260812 01:11:52.182574 30164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:11:52.189042 30164 webserver.cc:460] Webserver started at http://127.20.151.62:37669/ using document root <none> and password file <none>
I20260812 01:11:52.189780 30164 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:52.189888 30164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:52.190207 30164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:52.193608 30164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "dd24234a3b32442aad252694baec4a41"
format_stamp: "Formatted at 2026-08-12 01:11:52 on dist-test-slave-21b9"
server_key: "40c39e9cf18bf6fcc3aee6757f5d18c8"
server_key_iv: "1e6f5222b27923ab14fd05e9bffc3052"
server_key_version: "encryptionkey@0"
I20260812 01:11:52.194468 30164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "dd24234a3b32442aad252694baec4a41"
format_stamp: "Formatted at 2026-08-12 01:11:52 on dist-test-slave-21b9"
server_key: "40c39e9cf18bf6fcc3aee6757f5d18c8"
server_key_iv: "1e6f5222b27923ab14fd05e9bffc3052"
server_key_version: "encryptionkey@0"
I20260812 01:11:52.199815 30164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:11:52.203444 30187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:52.204872 30164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:52.205051 30164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "dd24234a3b32442aad252694baec4a41"
format_stamp: "Formatted at 2026-08-12 01:11:52 on dist-test-slave-21b9"
server_key: "40c39e9cf18bf6fcc3aee6757f5d18c8"
server_key_iv: "1e6f5222b27923ab14fd05e9bffc3052"
server_key_version: "encryptionkey@0"
I20260812 01:11:52.205273 30164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:52.226286 30164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:52.227170 30164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:52.227506 30164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:52.245929 30164 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:38487
I20260812 01:11:52.245973 30271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:38487 every 8 connection(s)
I20260812 01:11:52.247313 30164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:52.248080 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 30164
I20260812 01:11:52.248358 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:52.249198 21084 external_mini_cluster.cc:1428] Setting key 6ae9b4b6dba1dcd6e984cc5f557732e2
I20260812 01:11:52.252048 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:52.262466 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:52.251127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.20.151.62:38487)
0812 01:11:52.252166 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:52.252182 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:52.252232 (+ 50us) server_negotiation.cc:407] Connection header received
0812 01:11:52.253893 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:52.253914 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:52.254353 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:52.254773 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:52.255363 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:52.256732 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:52.258060 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:52.258718 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:52.258886 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:52.260122 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:52.260133 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:52.260150 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:52.260740 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:52.260846 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:52.260867 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:52.260975 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:52.261205 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 01:11:52.261521 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":367,"threads_started":1}
I20260812 01:11:52.262928 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41: Bootstrap starting.
I20260812 01:11:52.266680 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:11:52.266794 30272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:52.267876 30272 log.cc:824] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:52.271147 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41: No bootstrap required, opened a new log
I20260812 01:11:52.279537 30272 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd24234a3b32442aad252694baec4a41" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38487 } }
I20260812 01:11:52.280084 30272 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:52.280213 30272 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd24234a3b32442aad252694baec4a41, State: Initialized, Role: FOLLOWER
I20260812 01:11:52.280928 30272 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd24234a3b32442aad252694baec4a41" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38487 } }
I20260812 01:11:52.281361 30272 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:52.281545 30272 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:52.281739 30272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:52.284716 30272 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd24234a3b32442aad252694baec4a41" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38487 } }
I20260812 01:11:52.285261 30272 leader_election.cc:302] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd24234a3b32442aad252694baec4a41; no voters:
I20260812 01:11:52.285872 30272 leader_election.cc:288] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:52.286031 30280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:52.286763 30280 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [term 1 LEADER]: Becoming Leader. State: Replica: dd24234a3b32442aad252694baec4a41, State: Running, Role: LEADER
I20260812 01:11:52.287364 30280 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd24234a3b32442aad252694baec4a41" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38487 } }
I20260812 01:11:52.288039 30272 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:52.290011 30283 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd24234a3b32442aad252694baec4a41. Latest consensus state: current_term: 1 leader_uuid: "dd24234a3b32442aad252694baec4a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd24234a3b32442aad252694baec4a41" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38487 } } }
I20260812 01:11:52.290346 30283 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:52.290551 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd24234a3b32442aad252694baec4a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd24234a3b32442aad252694baec4a41" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38487 } } }
I20260812 01:11:52.290953 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:52.292783 30291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:52.298425 30291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:52.307520 30291 catalog_manager.cc:1347] Generated new cluster ID: d2ceb93a9bd34d01a11f32e33920dcc2
I20260812 01:11:52.307669 30291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:52.319129 30291 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:52.320600 30291 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:52.330960 30291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd24234a3b32442aad252694baec4a41: Generated new TSK 0
I20260812 01:11:52.331910 30291 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 01:11:52.377920 30317 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 01:11:52.378127 30317 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 01:11:52.380597 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:52.356688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.20.151.62:38487)
0812 01:11:52.356902 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:52.356909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:52.357350 (+ 441us) server_negotiation.cc:407] Connection header received
0812 01:11:52.357704 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:52.357710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:52.357802 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:52.357956 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0812 01:11:52.358559 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:52.359776 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:52.361218 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:52.361921 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:52.362066 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:52.362597 (+ 531us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 01:11:52.379882 (+ 17285us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 01:11:52.380259 (+ 377us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 01:11:52.380743 30277 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 01:11:52.382043 30316 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:11:52.356537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:38487 (local address 127.0.0.1:50600)
0812 01:11:52.357241 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:52.357264 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:52.357412 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:52.357973 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:52.357976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:52.357987 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0812 01:11:52.358367 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:52.358382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:52.359924 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:52.359928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:52.361078 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:52.361091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:52.362093 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:52.362098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:52.362224 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:52.362284 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 01:11:52.380159 (+ 17875us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 01:11:52.380435 (+ 276us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.151.62:38487: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":384,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.151.62:38487: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 01:11:52.384016 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 30164
I20260812 01:11:52.385093 30266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:52.481058 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 30164
2026-08-12T01:11:52Z chronyd exiting
I20260812 01:11:52.538372 21084 test_util.cc:175] -----------------------------------------------
I20260812 01:11:52.538518 21084 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786497066589499-21084-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (784 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 01:11:52.542390 21084 mini_oidc.cc:152] Starting JWKS server
I20260812 01:11:52.544252 21084 webserver.cc:460] Webserver started at https://127.0.0.1:33555/ using document root <none> and password file <none>
I20260812 01:11:52.544469 21084 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 01:11:52.545120 21084 webserver.cc:460] Webserver started at http://127.0.0.1:39581/ using document root <none> and password file <none>
2026-08-12T01:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:52Z Disabled control of system clock
I20260812 01:11:52.582826 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:45237
--webserver_interface=127.20.151.62
--webserver_port=0
--builtin_ntp_servers=127.20.151.20:32843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:45237
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33555/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/testchainca.cert with env {}
W20260812 01:11:52.926613 30324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:52.926949 30324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:52.927095 30324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:52.934610 30324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:52.934748 30324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:52.934830 30324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:52.934907 30324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:52.934970 30324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 01:11:52.935060 30324 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 01:11:52.935118 30324 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33555/jwks.json
I20260812 01:11:52.945477 30324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:32843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:45237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:45237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33555/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:52.946898 30324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:52.948297 30324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:52.960389 30339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:52.960412 30336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:52.960389 30333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:52.961187 30324 server_base.cc:1034] running on GCE node
I20260812 01:11:52.961999 30324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:52.963472 30324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:52.964689 30324 hybrid_clock.cc:648] HybridClock initialized: now 1786497112964638 us; error 49 us; skew 500 ppm
I20260812 01:11:52.965024 30324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:11:52.967353 30324 webserver.cc:460] Webserver started at http://127.20.151.62:46123/ using document root <none> and password file <none>
I20260812 01:11:52.968051 30324 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:52.968161 30324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:52.968482 30324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:52.971509 30324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "e0228382064d4b89ac1d6a2685a67210"
format_stamp: "Formatted at 2026-08-12 01:11:52 on dist-test-slave-21b9"
server_key: "b59f78a3ad5ecad07267df9d8cbb1b6b"
server_key_iv: "954657a39b5024d9b0e548d0b141149e"
server_key_version: "encryptionkey@0"
I20260812 01:11:52.972330 30324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "e0228382064d4b89ac1d6a2685a67210"
format_stamp: "Formatted at 2026-08-12 01:11:52 on dist-test-slave-21b9"
server_key: "b59f78a3ad5ecad07267df9d8cbb1b6b"
server_key_iv: "954657a39b5024d9b0e548d0b141149e"
server_key_version: "encryptionkey@0"
I20260812 01:11:52.977541 30324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:11:52.981132 30355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:52.982422 30324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:11:52.982600 30324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "e0228382064d4b89ac1d6a2685a67210"
format_stamp: "Formatted at 2026-08-12 01:11:52 on dist-test-slave-21b9"
server_key: "b59f78a3ad5ecad07267df9d8cbb1b6b"
server_key_iv: "954657a39b5024d9b0e548d0b141149e"
server_key_version: "encryptionkey@0"
I20260812 01:11:52.982842 30324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:53.007519 30324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:53.008605 30324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:53.008925 30324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:53.027822 30324 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:45237
I20260812 01:11:53.027829 30448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:45237 every 8 connection(s)
I20260812 01:11:53.029398 30324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:53.033470 30449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:53.037019 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 30324
I20260812 01:11:53.037241 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:53.038085 21084 external_mini_cluster.cc:1428] Setting key 9fb552898774e0fa584df5b7a6913141
I20260812 01:11:53.046911 30449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210: Bootstrap starting.
I20260812 01:11:53.050374 30449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:53.051668 30449 log.cc:824] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:53.052481 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:53.040334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.20.151.62:45237)
0812 01:11:53.041400 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:53.041412 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:53.041454 (+ 42us) server_negotiation.cc:407] Connection header received
0812 01:11:53.043199 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:53.043226 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:53.043825 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:53.044256 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:53.044910 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.046242 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.047712 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.048421 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.048608 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:53.049791 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:53.049799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:53.049815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:53.050413 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:53.050545 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:53.050575 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:53.050706 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:53.050964 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 01:11:53.051314 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":381,"threads_started":1}
I20260812 01:11:53.054967 30449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210: No bootstrap required, opened a new log
I20260812 01:11:53.056795 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:11:53.064002 30449 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0228382064d4b89ac1d6a2685a67210" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45237 } }
I20260812 01:11:53.064380 30449 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:53.064471 30449 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0228382064d4b89ac1d6a2685a67210, State: Initialized, Role: FOLLOWER
I20260812 01:11:53.065042 30449 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0228382064d4b89ac1d6a2685a67210" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45237 } }
I20260812 01:11:53.065263 30449 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:53.065364 30449 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:53.065500 30449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:53.068043 30449 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0228382064d4b89ac1d6a2685a67210" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45237 } }
I20260812 01:11:53.068563 30449 leader_election.cc:302] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0228382064d4b89ac1d6a2685a67210; no voters:
I20260812 01:11:53.069181 30449 leader_election.cc:288] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:53.069339 30457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:53.070115 30457 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [term 1 LEADER]: Becoming Leader. State: Replica: e0228382064d4b89ac1d6a2685a67210, State: Running, Role: LEADER
I20260812 01:11:53.070636 30457 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0228382064d4b89ac1d6a2685a67210" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45237 } }
I20260812 01:11:53.071202 30449 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:53.073233 30460 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0228382064d4b89ac1d6a2685a67210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0228382064d4b89ac1d6a2685a67210" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45237 } } }
I20260812 01:11:53.073622 30460 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:53.074193 30462 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0228382064d4b89ac1d6a2685a67210. Latest consensus state: current_term: 1 leader_uuid: "e0228382064d4b89ac1d6a2685a67210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0228382064d4b89ac1d6a2685a67210" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45237 } } }
I20260812 01:11:53.074518 30462 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:53.076015 30469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:53.081637 30469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:53.090968 30469 catalog_manager.cc:1347] Generated new cluster ID: 88726c93d14e43768aed790123a4d76b
I20260812 01:11:53.091104 30469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:53.115392 30469 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:53.116740 30469 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:53.132179 30469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0228382064d4b89ac1d6a2685a67210: Generated new TSK 0
I20260812 01:11:53.133318 30469 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 01:11:53.182106 30504 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 01:11:53.182255 30504 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 01:11:53.183418 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:53.156771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.20.151.62:45237)
0812 01:11:53.156989 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:53.156996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:53.157318 (+ 322us) server_negotiation.cc:407] Connection header received
0812 01:11:53.157559 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:53.157564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:53.157647 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:53.157783 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0812 01:11:53.158291 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.159015 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.160386 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.160818 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.160927 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:53.161438 (+ 511us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 01:11:53.182646 (+ 21208us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33555/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 01:11:53.183083 (+ 437us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33555/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 01:11:53.183573 30454 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33555/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 01:11:53.183660 30503 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:11:53.156623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:45237 (local address 127.0.0.1:39860)
0812 01:11:53.157216 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:53.157234 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:53.157383 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:53.157816 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:53.157821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:53.157833 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0812 01:11:53.158125 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:53.158134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.159153 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.159157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:53.160243 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:53.160255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.160990 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.160997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:53.161145 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:53.161181 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 01:11:53.182926 (+ 21745us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33555/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 01:11:53.183230 (+ 304us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.151.62:45237: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33555/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":387,"threads_started":1}
I20260812 01:11:53.184988 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 30324
I20260812 01:11:53.186149 30443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:53.282666 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 30324
2026-08-12T01:11:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (806 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 01:11:53.348625 21084 mini_oidc.cc:152] Starting JWKS server
I20260812 01:11:53.350584 21084 webserver.cc:460] Webserver started at https://127.0.0.1:36169/ using document root <none> and password file <none>
I20260812 01:11:53.350822 21084 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 01:11:53.351522 21084 webserver.cc:460] Webserver started at http://127.0.0.1:32801/ using document root <none> and password file <none>
2026-08-12T01:11:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:53Z Disabled control of system clock
I20260812 01:11:53.390190 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:45789
--webserver_interface=127.20.151.62
--webserver_port=0
--builtin_ntp_servers=127.20.151.20:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:45789
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36169/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 01:11:53.734586 30517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:53.734910 30517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:53.735039 30517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:53.742540 30517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:53.742666 30517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:53.742705 30517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:53.742781 30517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:53.742841 30517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 01:11:53.742931 30517 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 01:11:53.742985 30517 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36169/jwks.json
I20260812 01:11:53.752629 30517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:45789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:45789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36169/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:53.753983 30517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:53.755856 30517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:53.767853 30526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:53.767931 30527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:53.767892 30530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:53.768756 30517 server_base.cc:1034] running on GCE node
I20260812 01:11:53.769583 30517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:53.771103 30517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:53.772307 30517 hybrid_clock.cc:648] HybridClock initialized: now 1786497113772263 us; error 61 us; skew 500 ppm
I20260812 01:11:53.772671 30517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:11:53.775028 30517 webserver.cc:460] Webserver started at http://127.20.151.62:43263/ using document root <none> and password file <none>
I20260812 01:11:53.775820 30517 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:53.775918 30517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:53.776257 30517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:53.779242 30517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "047d7eeb08994e77b34ec7e2d1a8be5b"
format_stamp: "Formatted at 2026-08-12 01:11:53 on dist-test-slave-21b9"
server_key: "bb988e4e14ae76a2da12dcdced37b5db"
server_key_iv: "61df8e5b102da6288effdfe41c448e0a"
server_key_version: "encryptionkey@0"
I20260812 01:11:53.780099 30517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "047d7eeb08994e77b34ec7e2d1a8be5b"
format_stamp: "Formatted at 2026-08-12 01:11:53 on dist-test-slave-21b9"
server_key: "bb988e4e14ae76a2da12dcdced37b5db"
server_key_iv: "61df8e5b102da6288effdfe41c448e0a"
server_key_version: "encryptionkey@0"
I20260812 01:11:53.785357 30517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:11:53.788899 30538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:53.790302 30517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:11:53.790478 30517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "047d7eeb08994e77b34ec7e2d1a8be5b"
format_stamp: "Formatted at 2026-08-12 01:11:53 on dist-test-slave-21b9"
server_key: "bb988e4e14ae76a2da12dcdced37b5db"
server_key_iv: "61df8e5b102da6288effdfe41c448e0a"
server_key_version: "encryptionkey@0"
I20260812 01:11:53.790714 30517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:53.822181 30517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:53.823112 30517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:53.823477 30517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:53.841882 30517 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:45789
I20260812 01:11:53.841889 30634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:45789 every 8 connection(s)
I20260812 01:11:53.843313 30517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:53.846864 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 30517
I20260812 01:11:53.847092 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:53.847327 30636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:53.847999 21084 external_mini_cluster.cc:1428] Setting key 91b2a4643e845c88f038f6f6c71d9ff1
I20260812 01:11:53.860829 30636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b: Bootstrap starting.
I20260812 01:11:53.861752 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:53.849976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.20.151.62:45789)
0812 01:11:53.851164 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:53.851187 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:53.851231 (+ 44us) server_negotiation.cc:407] Connection header received
0812 01:11:53.853044 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:53.853066 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:53.853537 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:53.853908 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:53.854500 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.855875 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.857189 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.857893 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.858078 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:53.859358 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:53.859368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:53.859383 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:53.859960 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:53.860042 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:53.860065 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:53.860176 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:53.860413 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 01:11:53.860735 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":453,"threads_started":1}
I20260812 01:11:53.864485 30636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:53.865641 30636 log.cc:824] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:53.866189 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:11:53.868660 30636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b: No bootstrap required, opened a new log
I20260812 01:11:53.876453 30636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047d7eeb08994e77b34ec7e2d1a8be5b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45789 } }
I20260812 01:11:53.876856 30636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:53.876950 30636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047d7eeb08994e77b34ec7e2d1a8be5b, State: Initialized, Role: FOLLOWER
I20260812 01:11:53.877499 30636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047d7eeb08994e77b34ec7e2d1a8be5b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45789 } }
I20260812 01:11:53.877728 30636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:53.877866 30636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:53.878019 30636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:53.880296 30636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047d7eeb08994e77b34ec7e2d1a8be5b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45789 } }
I20260812 01:11:53.880818 30636 leader_election.cc:302] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 047d7eeb08994e77b34ec7e2d1a8be5b; no voters:
I20260812 01:11:53.881394 30636 leader_election.cc:288] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:53.881528 30643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:53.882714 30643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [term 1 LEADER]: Becoming Leader. State: Replica: 047d7eeb08994e77b34ec7e2d1a8be5b, State: Running, Role: LEADER
I20260812 01:11:53.883231 30643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047d7eeb08994e77b34ec7e2d1a8be5b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45789 } }
I20260812 01:11:53.883597 30636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:53.886152 30645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "047d7eeb08994e77b34ec7e2d1a8be5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047d7eeb08994e77b34ec7e2d1a8be5b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45789 } } }
I20260812 01:11:53.886595 30645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:53.886180 30647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 047d7eeb08994e77b34ec7e2d1a8be5b. Latest consensus state: current_term: 1 leader_uuid: "047d7eeb08994e77b34ec7e2d1a8be5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047d7eeb08994e77b34ec7e2d1a8be5b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45789 } } }
I20260812 01:11:53.887526 30647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:53.888262 30656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:53.892899 30656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:53.901922 30656 catalog_manager.cc:1347] Generated new cluster ID: 45f3fb095d524cb182c7c4704fa776aa
I20260812 01:11:53.902050 30656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:53.930088 30656 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:53.932062 30656 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:53.938978 30656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 047d7eeb08994e77b34ec7e2d1a8be5b: Generated new TSK 0
I20260812 01:11:53.940092 30656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:11:53.968495 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:53.959902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.20.151.62:45789)
0812 01:11:53.960117 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:53.960123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:53.960444 (+ 321us) server_negotiation.cc:407] Connection header received
0812 01:11:53.960682 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:53.960688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:53.960784 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:53.960970 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0812 01:11:53.961391 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.962564 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.964008 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.964710 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.964832 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:53.965293 (+ 461us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 01:11:53.967807 (+ 2514us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 01:11:53.968156 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49696: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 01:11:53.968643 30640 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49696: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 01:11:53.970196 30676 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:11:53.959752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:45789 (local address 127.0.0.1:49696)
0812 01:11:53.960351 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:53.960367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:53.960491 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:53.960965 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:53.960969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:53.960977 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 01:11:53.961250 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:53.961257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.962737 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.962741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:53.963864 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:53.963877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:53.964872 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:53.964876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:53.964999 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:53.965031 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 01:11:53.968034 (+ 3003us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 01:11:53.969755 (+ 1721us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.151.62:45789: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36169/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":1415,"thread_start_us":319,"threads_started":1}
I20260812 01:11:53.972242 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 30517
I20260812 01:11:53.973542 30630 generic_service.cc:196] Checking for leaks (request via RPC)
E20260812 01:11:53.982712 30680 webserver.cc:533] Webserver: sslize failed:
E20260812 01:11:53.982853 30680 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 01:11:54.073949 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 30517
2026-08-12T01:11:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (789 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T01:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:54Z Disabled control of system clock
I20260812 01:11:54.178508 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43915
--webserver_interface=127.20.151.62
--webserver_port=0
--builtin_ntp_servers=127.20.151.20:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:43915
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/testchainca.cert with env {}
W20260812 01:11:54.537241 30694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:54.537573 30694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:54.537724 30694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:54.545092 30694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:54.545212 30694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:54.545276 30694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:54.545351 30694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:54.545408 30694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 01:11:54.545493 30694 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 01:11:54.545539 30694 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 01:11:54.555905 30694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:43915
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:54.557366 30694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:54.558678 30694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:54.571063 30705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:54.571012 30703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:54.571012 30707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:54.572002 30694 server_base.cc:1034] running on GCE node
I20260812 01:11:54.572784 30694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:54.574235 30694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:54.575453 30694 hybrid_clock.cc:648] HybridClock initialized: now 1786497114575392 us; error 42 us; skew 500 ppm
I20260812 01:11:54.575884 30694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:11:54.578343 30694 webserver.cc:460] Webserver started at http://127.20.151.62:39543/ using document root <none> and password file <none>
I20260812 01:11:54.579092 30694 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:54.579205 30694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:54.579576 30694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:54.582549 30694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "780c97b71cdd4364959265f2f38dec54"
format_stamp: "Formatted at 2026-08-12 01:11:54 on dist-test-slave-21b9"
server_key: "a9e0df17ce015353cf7af3e7221d3556"
server_key_iv: "8afb05c2e738513fcd3d29dff47458a6"
server_key_version: "encryptionkey@0"
I20260812 01:11:54.583393 30694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "780c97b71cdd4364959265f2f38dec54"
format_stamp: "Formatted at 2026-08-12 01:11:54 on dist-test-slave-21b9"
server_key: "a9e0df17ce015353cf7af3e7221d3556"
server_key_iv: "8afb05c2e738513fcd3d29dff47458a6"
server_key_version: "encryptionkey@0"
I20260812 01:11:54.588683 30694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260812 01:11:54.592233 30721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:54.593657 30694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260812 01:11:54.593839 30694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "780c97b71cdd4364959265f2f38dec54"
format_stamp: "Formatted at 2026-08-12 01:11:54 on dist-test-slave-21b9"
server_key: "a9e0df17ce015353cf7af3e7221d3556"
server_key_iv: "8afb05c2e738513fcd3d29dff47458a6"
server_key_version: "encryptionkey@0"
I20260812 01:11:54.594044 30694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:54.620349 30694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:54.621259 30694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:54.621556 30694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:54.639936 30694 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:43915
I20260812 01:11:54.639957 30806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:43915 every 8 connection(s)
I20260812 01:11:54.641327 30694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:54.645046 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 30694
I20260812 01:11:54.645222 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:54.645236 30807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:11:54.645871 21084 external_mini_cluster.cc:1428] Setting key 83caf53de42b7979e550d9cd08371f7c
I20260812 01:11:54.658797 30807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54: Bootstrap starting.
I20260812 01:11:54.660763 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:54.647774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.20.151.62:43915)
0812 01:11:54.648879 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:54.648890 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:54.648939 (+ 49us) server_negotiation.cc:407] Connection header received
0812 01:11:54.651195 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:54.651220 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:54.651787 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:54.652128 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:54.652722 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:54.653971 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:54.655958 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:54.656787 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:54.656975 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:54.658303 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:54.658311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:54.658326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:11:54.658886 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:54.659013 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:11:54.659042 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:54.659140 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:54.659432 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 01:11:54.659767 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":406,"threads_started":1}
I20260812 01:11:54.662494 30807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:54.663699 30807 log.cc:824] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:54.664770 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:11:54.666958 30807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54: No bootstrap required, opened a new log
I20260812 01:11:54.674635 30807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780c97b71cdd4364959265f2f38dec54" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43915 } }
I20260812 01:11:54.674995 30807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:54.675082 30807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780c97b71cdd4364959265f2f38dec54, State: Initialized, Role: FOLLOWER
I20260812 01:11:54.675715 30807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780c97b71cdd4364959265f2f38dec54" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43915 } }
I20260812 01:11:54.675942 30807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:54.676100 30807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:54.676257 30807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:54.678423 30807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780c97b71cdd4364959265f2f38dec54" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43915 } }
I20260812 01:11:54.678879 30807 leader_election.cc:302] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 780c97b71cdd4364959265f2f38dec54; no voters:
I20260812 01:11:54.679486 30807 leader_election.cc:288] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:54.679593 30815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:54.680629 30815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [term 1 LEADER]: Becoming Leader. State: Replica: 780c97b71cdd4364959265f2f38dec54, State: Running, Role: LEADER
I20260812 01:11:54.681149 30815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780c97b71cdd4364959265f2f38dec54" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43915 } }
I20260812 01:11:54.681679 30807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:54.683878 30817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 780c97b71cdd4364959265f2f38dec54. Latest consensus state: current_term: 1 leader_uuid: "780c97b71cdd4364959265f2f38dec54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780c97b71cdd4364959265f2f38dec54" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43915 } } }
I20260812 01:11:54.684258 30817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:54.685003 30816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "780c97b71cdd4364959265f2f38dec54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780c97b71cdd4364959265f2f38dec54" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43915 } } }
I20260812 01:11:54.685366 30816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:54.686041 30822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:54.690730 30822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:54.699492 30822 catalog_manager.cc:1347] Generated new cluster ID: cdfa28eb1d3c4d27a543dc003f5a38e9
I20260812 01:11:54.699612 30822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:54.726279 30822 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:54.728188 30822 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:54.735106 30822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 780c97b71cdd4364959265f2f38dec54: Generated new TSK 0
I20260812 01:11:54.735972 30822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:11:54.839082 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:54.754617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.20.151.62:43915)
0812 01:11:54.754832 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:54.754838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:54.755254 (+ 416us) server_negotiation.cc:407] Connection header received
0812 01:11:54.755508 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:54.755522 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:54.755631 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:54.755817 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0812 01:11:54.756241 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:54.757365 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:54.758760 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:54.759470 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:54.759606 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:54.760128 (+ 522us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 01:11:54.838456 (+ 78328us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 01:11:54.838826 (+ 370us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51868: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 01:11:54.839227 30812 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51868: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 01:11:54.839635 30855 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:11:54.754486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:43915 (local address 127.0.0.1:51868)
0812 01:11:54.755148 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:54.755165 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:54.755326 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:54.755814 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:54.755817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:54.755826 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0812 01:11:54.756095 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:54.756103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:54.757524 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:54.757528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:54.758617 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:54.758629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:54.759650 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:54.759654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:54.759793 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:54.759828 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 01:11:54.838787 (+ 78959us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 01:11:54.839158 (+ 371us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.151.62:43915: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":336,"threads_started":1}
I20260812 01:11:54.841274 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 30694
I20260812 01:11:54.842511 30801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:54.946213 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 30694
2026-08-12T01:11:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (851 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (431 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (432 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:11:55 dist-test-slave-21b9 krb5kdc[30887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:11:55 dist-test-slave-21b9 krb5kdc[30887](info): set up 2 sockets
Aug 12 01:11:55 dist-test-slave-21b9 krb5kdc[30887](info): commencing operation
krb5kdc: starting...
W20260812 01:11:57.091830 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.003s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:11:57 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:11:57.369418 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:36965
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:36965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:57.711027 30910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:57.711352 30910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:57.711496 30910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:57.718883 30910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:11:57.718994 30910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:57.719069 30910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:57.719148 30910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:11:57.719210 30910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:11:57.729174 30910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:36965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:36965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:57.730581 30910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:57.731956 30910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:57.744002 30924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:57.744002 30921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:57.744002 30919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:57.744762 30910 server_base.cc:1034] running on GCE node
I20260812 01:11:57.745579 30910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:57.746974 30910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:57.748185 30910 hybrid_clock.cc:648] HybridClock initialized: now 1786497117748140 us; error 25 us; skew 500 ppm
I20260812 01:11:57.748538 30910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:57 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497117, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:57.755096 30910 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:11:57.756686 30910 webserver.cc:460] Webserver started at http://127.20.151.62:40159/ using document root <none> and password file <none>
I20260812 01:11:57.757421 30910 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:57.757546 30910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:57.757856 30910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:57.760974 30910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "ac66cb21ebe34b56b9d6cd9da8944996"
format_stamp: "Formatted at 2026-08-12 01:11:57 on dist-test-slave-21b9"
server_key: "e3931b93f061e33292367670e9994366"
server_key_iv: "c0ab9272ade53e99d17577060996069a"
server_key_version: "encryptionkey@0"
I20260812 01:11:57.761765 30910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "ac66cb21ebe34b56b9d6cd9da8944996"
format_stamp: "Formatted at 2026-08-12 01:11:57 on dist-test-slave-21b9"
server_key: "e3931b93f061e33292367670e9994366"
server_key_iv: "c0ab9272ade53e99d17577060996069a"
server_key_version: "encryptionkey@0"
I20260812 01:11:57.767050 30910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:11:57.770656 30939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:57.772011 30910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:11:57.772192 30910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "ac66cb21ebe34b56b9d6cd9da8944996"
format_stamp: "Formatted at 2026-08-12 01:11:57 on dist-test-slave-21b9"
server_key: "e3931b93f061e33292367670e9994366"
server_key_iv: "c0ab9272ade53e99d17577060996069a"
server_key_version: "encryptionkey@0"
I20260812 01:11:57.772382 30910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:57.790266 30910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:57.794507 30910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:57.794818 30910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:57.813591 30910 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:36965
I20260812 01:11:57.813642 31028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:36965 every 8 connection(s)
I20260812 01:11:57.815052 30910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:11:57.815955 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 30910
I20260812 01:11:57.816192 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:11:57.816830 21084 external_mini_cluster.cc:1428] Setting key c9b931b9da4bc918b81c5c5ac3b3694c
I20260812 01:11:57.819423 31030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 01:11:57 dist-test-slave-21b9 krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:57.832337 31030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996: Bootstrap starting.
I20260812 01:11:57.835763 31030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996: Neither blocks nor log segments found. Creating new log.
I20260812 01:11:57.836902 31030 log.cc:824] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996: Log is configured to *not* fsync() on all Append() calls
I20260812 01:11:57.839798 31030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996: No bootstrap required, opened a new log
I20260812 01:11:57.839833 31034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:57.818865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.20.151.62:36965)
0812 01:11:57.819688 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:57.819700 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:57.819743 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:11:57.821403 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:57.821425 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:57.821802 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:57.822123 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:57.823956 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:57.825318 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:57.827038 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:57.827839 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:57.828006 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:57.832669 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:57.832677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:57.832692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:57.832729 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:57.835789 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:57.836488 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:57.836495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:57.836503 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:57.836679 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:57.837081 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:57.837085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:57.837087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:57.837637 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:57.837945 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:57.838377 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 01:11:57.838787 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":351,"threads_started":1}
I20260812 01:11:57.847849 31030 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66cb21ebe34b56b9d6cd9da8944996" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36965 } }
I20260812 01:11:57.848232 31030 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:11:57.848330 31030 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac66cb21ebe34b56b9d6cd9da8944996, State: Initialized, Role: FOLLOWER
I20260812 01:11:57.848863 31030 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66cb21ebe34b56b9d6cd9da8944996" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36965 } }
I20260812 01:11:57.849084 31030 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:11:57.849237 31030 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:11:57.849397 31030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:11:57.851676 31030 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66cb21ebe34b56b9d6cd9da8944996" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36965 } }
I20260812 01:11:57.852226 31030 leader_election.cc:302] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac66cb21ebe34b56b9d6cd9da8944996; no voters:
I20260812 01:11:57.852798 31030 leader_election.cc:288] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:11:57.852963 31036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:11:57.853667 31036 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [term 1 LEADER]: Becoming Leader. State: Replica: ac66cb21ebe34b56b9d6cd9da8944996, State: Running, Role: LEADER
I20260812 01:11:57.854171 31036 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66cb21ebe34b56b9d6cd9da8944996" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36965 } }
I20260812 01:11:57.854884 31030 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:11:57.857014 31037 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac66cb21ebe34b56b9d6cd9da8944996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66cb21ebe34b56b9d6cd9da8944996" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36965 } } }
I20260812 01:11:57.857797 31037 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:57.857205 31040 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac66cb21ebe34b56b9d6cd9da8944996. Latest consensus state: current_term: 1 leader_uuid: "ac66cb21ebe34b56b9d6cd9da8944996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66cb21ebe34b56b9d6cd9da8944996" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36965 } } }
I20260812 01:11:57.859611 31040 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996 [sys.catalog]: This master's current role is: LEADER
I20260812 01:11:57.859933 31054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:11:57.865283 31054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:11:57.875074 31054 catalog_manager.cc:1347] Generated new cluster ID: abdd4aa9a96c49e8a71aca12c42bafe5
I20260812 01:11:57.875195 31054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:11:57.885870 31054 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:11:57.887307 31054 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:11:57.905203 31054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac66cb21ebe34b56b9d6cd9da8944996: Generated new TSK 0
I20260812 01:11:57.906098 31054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:11:57.986552 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36965
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:58.335007 31079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:58.335330 31079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:58.335487 31079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:58.343166 31079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:58.343317 31079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:58.343495 31079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:11:58.353539 31079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:58.354938 31079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:58.356498 31079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:58.369519 31092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:58.369611 31095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:58.369679 31088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:58.370636 31079 server_base.cc:1034] running on GCE node
I20260812 01:11:58.371250 31079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:58.372231 31079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:58.373443 31079 hybrid_clock.cc:648] HybridClock initialized: now 1786497118373388 us; error 39 us; skew 500 ppm
I20260812 01:11:58.373797 31079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:58 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497118, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:58.376972 31079 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:11:58.378408 31079 webserver.cc:460] Webserver started at http://127.20.151.1:39807/ using document root <none> and password file <none>
I20260812 01:11:58.379050 31079 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:58.379154 31079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:58.379544 31079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:58.382916 31079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "d92f7d726f364ffe9456428a1cc5039c"
format_stamp: "Formatted at 2026-08-12 01:11:58 on dist-test-slave-21b9"
server_key: "77e5374a36e0afe56635d5ad0e502696"
server_key_iv: "419dc99882059b88de815f0d5d21c1fa"
server_key_version: "encryptionkey@0"
I20260812 01:11:58.383718 31079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "d92f7d726f364ffe9456428a1cc5039c"
format_stamp: "Formatted at 2026-08-12 01:11:58 on dist-test-slave-21b9"
server_key: "77e5374a36e0afe56635d5ad0e502696"
server_key_iv: "419dc99882059b88de815f0d5d21c1fa"
server_key_version: "encryptionkey@0"
I20260812 01:11:58.388655 31079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260812 01:11:58.392019 31115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:58.393276 31079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:58.393438 31079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "d92f7d726f364ffe9456428a1cc5039c"
format_stamp: "Formatted at 2026-08-12 01:11:58 on dist-test-slave-21b9"
server_key: "77e5374a36e0afe56635d5ad0e502696"
server_key_iv: "419dc99882059b88de815f0d5d21c1fa"
server_key_version: "encryptionkey@0"
I20260812 01:11:58.393671 31079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:58.416749 31079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:58.421255 31079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:58.421556 31079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:58.422585 31079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:58.424038 31079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:58.424125 31079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:58.424257 31079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:58.424314 31079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:58.456889 31079 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:40597
I20260812 01:11:58.456967 31311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:40597 every 8 connection(s)
I20260812 01:11:58.458209 31079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:11:58.465435 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 31079
I20260812 01:11:58.465647 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:11:58.466274 21084 external_mini_cluster.cc:1428] Setting key 5dcf1d601cca85cf4c1fff87247a0cbc
Aug 12 01:11:58 dist-test-slave-21b9 krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497118, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:58.479449 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:58.460861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:46105 (local address 127.20.151.62:36965)
0812 01:11:58.461306 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:58.461311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:58.462376 (+ 1065us) server_negotiation.cc:407] Connection header received
0812 01:11:58.464314 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:58.464319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:58.464404 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:58.464500 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:58.466163 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:58.466939 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:58.468926 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:58.469467 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:58.469607 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:58.473255 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:58.473262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:58.473267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:58.473303 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:58.476293 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:58.477069 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:58.477074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:58.477077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:58.477224 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:58.477667 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:58.477672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:58.477674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:58.477995 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:58.478136 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:58.478846 (+ 710us) server_negotiation.cc:299] Negotiation successful
0812 01:11:58.479059 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":235,"threads_started":1}
I20260812 01:11:58.480432 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:58.461432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36965 (local address 127.20.151.1:46105)
0812 01:11:58.462213 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:58.462252 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:58.464035 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:58.464757 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:58.464765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:58.465351 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:58.465978 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:58.466003 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:58.467099 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:58.467103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:58.468776 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:58.468790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:58.469672 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:58.469676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:58.469872 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:58.470513 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:58.470537 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:58.473014 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:58.476467 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:58.476474 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:58.476486 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:58.476940 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:58.477331 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:58.477334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:58.477336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:58.477559 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:58.478100 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:58.478106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:58.478430 (+ 324us) client_negotiation.cc:769] Sending connection context
0812 01:11:58.478688 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 01:11:58.479091 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":312,"threads_started":1}
I20260812 01:11:58.482122 31317 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36965
I20260812 01:11:58.482446 31317 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:58.483182 31317 heartbeater.cc:511] Master 127.20.151.62:36965 requested a full tablet report, sending...
I20260812 01:11:58.485574 30973 ts_manager.cc:194] Registered new tserver with Master: d92f7d726f364ffe9456428a1cc5039c (127.20.151.1:40597)
I20260812 01:11:58.487507 30973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:46105
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:11:58.528079 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36965
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:58.871731 31330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:58.872043 31330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:58.872192 31330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:58.879629 31330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:58.879766 31330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:58.879920 31330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:11:58.890076 31330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:58.891435 31330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:58.892795 31330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:58.906035 31341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:58.906038 31338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:58.906153 31337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:58.906878 31330 server_base.cc:1034] running on GCE node
I20260812 01:11:58.907614 31330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:58.908623 31330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:58.909825 31330 hybrid_clock.cc:648] HybridClock initialized: now 1786497118909776 us; error 39 us; skew 500 ppm
I20260812 01:11:58.910154 31330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:58 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497118, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:58.913409 31330 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:11:58.914942 31330 webserver.cc:460] Webserver started at http://127.20.151.2:33395/ using document root <none> and password file <none>
I20260812 01:11:58.915694 31330 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:58.915812 31330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:58.916126 31330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:58.919183 31330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "cf0f02aba4be47b48c615eabf8cf6686"
format_stamp: "Formatted at 2026-08-12 01:11:58 on dist-test-slave-21b9"
server_key: "0c361196eba8a5439898546fca6b273b"
server_key_iv: "c330059a11d934a6069e1dd6fa9fdc5d"
server_key_version: "encryptionkey@0"
I20260812 01:11:58.920136 31330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "cf0f02aba4be47b48c615eabf8cf6686"
format_stamp: "Formatted at 2026-08-12 01:11:58 on dist-test-slave-21b9"
server_key: "0c361196eba8a5439898546fca6b273b"
server_key_iv: "c330059a11d934a6069e1dd6fa9fdc5d"
server_key_version: "encryptionkey@0"
I20260812 01:11:58.925508 31330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:11:58.929292 31351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:58.930678 31330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:11:58.930847 31330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "cf0f02aba4be47b48c615eabf8cf6686"
format_stamp: "Formatted at 2026-08-12 01:11:58 on dist-test-slave-21b9"
server_key: "0c361196eba8a5439898546fca6b273b"
server_key_iv: "c330059a11d934a6069e1dd6fa9fdc5d"
server_key_version: "encryptionkey@0"
I20260812 01:11:58.931064 31330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:58.949620 31330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:58.953701 31330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:58.953991 31330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:58.955053 31330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:58.956571 31330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:58.956655 31330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:58.956779 31330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:58.956831 31330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:58.988613 31330 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:41053
I20260812 01:11:58.988688 31549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:41053 every 8 connection(s)
I20260812 01:11:58.989910 31330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:11:58.994998 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 31330
I20260812 01:11:58.995172 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:11:58.995630 21084 external_mini_cluster.cc:1428] Setting key 261c3bbcc1828f69b2b27e45e0410d11
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497118, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:59.012889 31552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:58.992747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:40811 (local address 127.20.151.62:36965)
0812 01:11:58.993192 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:58.993196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:58.994098 (+ 902us) server_negotiation.cc:407] Connection header received
0812 01:11:58.996100 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:58.996106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:58.996206 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:58.996323 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:58.998368 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:58.999099 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.001321 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.001788 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.001895 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:59.005958 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:59.005965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:59.005970 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:59.006012 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:59.009295 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.010287 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.010292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.010294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.010458 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.011009 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.011012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.011014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.011336 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:59.011493 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:59.012394 (+ 901us) server_negotiation.cc:299] Negotiation successful
0812 01:11:59.012619 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":235,"threads_started":1}
I20260812 01:11:59.014310 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:58.993168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36965 (local address 127.20.151.2:40811)
0812 01:11:58.993915 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:58.993965 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:58.995816 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:58.996675 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:58.996687 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:58.997335 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:58.998183 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:58.998212 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:58.999290 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:58.999296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:59.001156 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:59.001173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.002033 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.002040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:59.002309 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:59.003056 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:59.003093 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:59.005689 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:59.009452 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:59.009463 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:59.009479 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:59.010128 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:59.010610 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:59.010615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:59.010618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:59.010894 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:59.011466 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:59.011475 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:59.011950 (+ 475us) client_negotiation.cc:769] Sending connection context
0812 01:11:59.012276 (+ 326us) client_negotiation.cc:240] Negotiation successful
0812 01:11:59.012848 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":290,"threads_started":1}
I20260812 01:11:59.015933 31550 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36965
I20260812 01:11:59.016247 31550 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:59.017050 31550 heartbeater.cc:511] Master 127.20.151.62:36965 requested a full tablet report, sending...
I20260812 01:11:59.018635 30973 ts_manager.cc:194] Registered new tserver with Master: cf0f02aba4be47b48c615eabf8cf6686 (127.20.151.2:41053)
I20260812 01:11:59.019502 30973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:40811
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:11:59.059955 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36965
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:11:59.398739 31562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:11:59.399036 31562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:11:59.399142 31562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:11:59.406414 31562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:11:59.406529 31562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:11:59.406648 31562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:11:59.416854 31562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:11:59.418241 31562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:11:59.419713 31562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:11:59.433334 31574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:59.433387 31580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:11:59.433454 31573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:11:59.434270 31562 server_base.cc:1034] running on GCE node
I20260812 01:11:59.435053 31562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:11:59.436182 31562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:11:59.437405 31562 hybrid_clock.cc:648] HybridClock initialized: now 1786497119437342 us; error 49 us; skew 500 ppm
I20260812 01:11:59.437769 31562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:59.441111 31562 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:11:59.442590 31562 webserver.cc:460] Webserver started at http://127.20.151.3:33597/ using document root <none> and password file <none>
I20260812 01:11:59.443338 31562 fs_manager.cc:359] Metadata directory not provided
I20260812 01:11:59.443482 31562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:11:59.443791 31562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:11:59.446796 31562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "f2bc5c7309f54fcb9997f07b60415dc1"
format_stamp: "Formatted at 2026-08-12 01:11:59 on dist-test-slave-21b9"
server_key: "410686492abcc4e5402ea234c39e6211"
server_key_iv: "6c1ea05444ae4be7a523a3d1bfcbbd1c"
server_key_version: "encryptionkey@0"
I20260812 01:11:59.447631 31562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "f2bc5c7309f54fcb9997f07b60415dc1"
format_stamp: "Formatted at 2026-08-12 01:11:59 on dist-test-slave-21b9"
server_key: "410686492abcc4e5402ea234c39e6211"
server_key_iv: "6c1ea05444ae4be7a523a3d1bfcbbd1c"
server_key_version: "encryptionkey@0"
I20260812 01:11:59.452903 31562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:11:59.456658 31595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:59.458107 31562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:11:59.458271 31562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "f2bc5c7309f54fcb9997f07b60415dc1"
format_stamp: "Formatted at 2026-08-12 01:11:59 on dist-test-slave-21b9"
server_key: "410686492abcc4e5402ea234c39e6211"
server_key_iv: "6c1ea05444ae4be7a523a3d1bfcbbd1c"
server_key_version: "encryptionkey@0"
I20260812 01:11:59.458494 31562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:11:59.488835 31562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:11:59.490533 31317 heartbeater.cc:503] Master 127.20.151.62:36965 was elected leader, sending a full tablet report...
I20260812 01:11:59.493613 31562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:11:59.493896 31562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:11:59.494863 31562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:11:59.496343 31562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:11:59.496416 31562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:59.496539 31562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:11:59.496631 31562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:11:59.527052 31562 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:42519
I20260812 01:11:59.527102 31790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:42519 every 8 connection(s)
I20260812 01:11:59.528366 31562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:11:59.537315 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 31562
I20260812 01:11:59.537483 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:11:59.537916 21084 external_mini_cluster.cc:1428] Setting key 6b2cac630096eecf6a04881ee9b4483b
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:11:59.548620 31793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:59.530963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:57621 (local address 127.20.151.62:36965)
0812 01:11:59.531437 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:59.531442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:59.532354 (+ 912us) server_negotiation.cc:407] Connection header received
0812 01:11:59.534244 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:59.534250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:59.534331 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:59.534429 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:59.536137 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.536938 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.538681 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.539201 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.539395 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:59.542974 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:59.542979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:59.542983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:59.543021 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:59.545562 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.546278 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.546282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.546285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.546411 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.546834 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.546839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.546842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.547116 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:59.547249 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:59.547996 (+ 747us) server_negotiation.cc:299] Negotiation successful
0812 01:11:59.548235 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":239,"threads_started":1}
I20260812 01:11:59.549587 31794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:59.531456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36965 (local address 127.20.151.3:57621)
0812 01:11:59.532191 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0812 01:11:59.532231 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 01:11:59.533968 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:11:59.534691 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:11:59.534700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:11:59.535342 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:11:59.535952 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:59.535979 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.537103 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.537106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:59.538489 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:11:59.538506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.539478 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.539484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:11:59.539728 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:59.540428 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:11:59.540458 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:11:59.542781 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:11:59.545707 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:59.545716 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:59.545733 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:59.546165 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:59.546516 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:11:59.546522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:11:59.546526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:11:59.546729 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:11:59.547244 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:11:59.547251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:11:59.547638 (+ 387us) client_negotiation.cc:769] Sending connection context
0812 01:11:59.547896 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 01:11:59.548434 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":305,"threads_started":1}
I20260812 01:11:59.550967 31791 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36965
I20260812 01:11:59.551313 31791 heartbeater.cc:464] Registering TS with master...
I20260812 01:11:59.552140 31791 heartbeater.cc:511] Master 127.20.151.62:36965 requested a full tablet report, sending...
I20260812 01:11:59.553577 30973 ts_manager.cc:194] Registered new tserver with Master: f2bc5c7309f54fcb9997f07b60415dc1 (127.20.151.3:42519)
I20260812 01:11:59.554410 30973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:57621
I20260812 01:11:59.565687 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/test-admin.keytab.
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:59.600729 21084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 01:11:59.605887 31806 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:11:59.603431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.20.151.1:40597)
0812 01:11:59.604140 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:59.604148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:59.604174 (+ 26us) server_negotiation.cc:407] Connection header received
0812 01:11:59.604242 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:59.604248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:59.604449 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:59.604610 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:59.605503 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49946: BlockingRecv error: recv got EOF from 127.0.0.1:49946 (error 108)
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":330,"threads_started":1}
W20260812 01:11:59.606143 21084 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5cc' of type FILE
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:59.608096 21084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:59.609426 21084 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:59.624300 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:59.611187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.20.151.1:40597)
0812 01:11:59.611407 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:59.611412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:59.611504 (+ 92us) server_negotiation.cc:407] Connection header received
0812 01:11:59.611659 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:59.611663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:59.611726 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:59.611808 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:59.612768 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.613831 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.615050 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.615705 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.615812 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:59.619139 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:59.619151 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:59.619167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:59.619210 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:59.621922 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.622520 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.622526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.622529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.622652 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.622926 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.622929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.622931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.623264 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:59.623553 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:59.623762 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 01:11:59.623989 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260812 01:11:59.630210 31806 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:11:59.628488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.20.151.1:40597)
0812 01:11:59.628681 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:59.628686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:59.628906 (+ 220us) server_negotiation.cc:407] Connection header received
0812 01:11:59.629051 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:59.629054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:59.629117 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:59.629195 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:59.629986 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49962: BlockingRecv error: recv got EOF from 127.0.0.1:49962 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 01:11:59.630759 21084 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5cc)
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:59.632154 21084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:59.633520 21084 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:59.647580 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:59.635260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.20.151.1:40597)
0812 01:11:59.635451 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:59.635456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:59.636245 (+ 789us) server_negotiation.cc:407] Connection header received
0812 01:11:59.636292 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:59.636295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:59.636345 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:59.636409 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:59.637331 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.637997 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.639051 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.639529 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.639623 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:59.642898 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:59.642903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:59.642907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:59.642947 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:59.645379 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.645986 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.645990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.645992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.646103 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.646474 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.646476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.646478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.646729 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:59.646934 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:59.647136 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 01:11:59.647341 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 01:11:59.651796 31806 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:11:59.650037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.20.151.1:40597)
0812 01:11:59.650239 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:59.650245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:59.650425 (+ 180us) server_negotiation.cc:407] Connection header received
0812 01:11:59.650552 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:59.650556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:59.650625 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:59.650712 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:59.651558 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49986: BlockingRecv error: recv got EOF from 127.0.0.1:49986 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260812 01:11:59.652237 21084 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786497066589499-21084-0/krb5kdc/krb5cc' of type FILE
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:59.653601 21084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:11:59.654762 21084 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:11:59.668749 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:59.656166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.20.151.1:40597)
0812 01:11:59.656331 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:59.656341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:59.656668 (+ 327us) server_negotiation.cc:407] Connection header received
0812 01:11:59.656824 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:59.656828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:59.656879 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:59.656952 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:59.657885 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.658580 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.659870 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.660315 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.660416 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:59.663891 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:59.663896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:59.663899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:59.663929 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:59.666316 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.666923 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.666926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.666929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.667058 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.667442 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.667445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.667448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.667700 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:59.667970 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:59.668255 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 01:11:59.668465 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 01:11:59.670149 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 31079
I20260812 01:11:59.680979 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:59.670838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50006 (local address 127.20.151.1:40597)
0812 01:11:59.671061 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:59.671066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:59.671251 (+ 185us) server_negotiation.cc:407] Connection header received
0812 01:11:59.671423 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:59.671426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:59.671479 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:59.671549 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:59.672367 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.673016 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.674147 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.674524 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.674606 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:59.676554 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:59.676558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:59.676561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:59.676589 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:59.678927 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.679479 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.679483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.679485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.679616 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.679888 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.679891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.679893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.680129 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:59.680318 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:59.680504 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 01:11:59.680658 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:11:59.681572 31306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:59.765448 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 31079
I20260812 01:11:59.792796 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 31330
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:11:59.808784 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:59.793720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.20.151.2:41053)
0812 01:11:59.794235 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:59.794243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:59.794268 (+ 25us) server_negotiation.cc:407] Connection header received
0812 01:11:59.794331 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:59.794337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:59.794538 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:59.794703 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:59.795801 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.796847 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.798133 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.798704 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.798819 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:59.802870 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:59.802883 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:59.802898 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:59.802939 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:59.805896 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.806601 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.806609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.806612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.806760 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.807130 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.807139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.807142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.807568 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:59.807868 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:59.808196 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 01:11:59.808459 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":245,"threads_started":1}
I20260812 01:11:59.810125 31539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:11:59.893468 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 31330
I20260812 01:11:59.916707 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 31562
Aug 12 01:11:59 dist-test-slave-21b9 krb5kdc[30887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:11:59.932765 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:11:59.917767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.20.151.3:42519)
0812 01:11:59.918458 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0812 01:11:59.918465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:11:59.918494 (+ 29us) server_negotiation.cc:407] Connection header received
0812 01:11:59.918558 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:11:59.918567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:11:59.918783 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:11:59.918934 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:11:59.920150 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.921257 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.922604 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:11:59.923215 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:11:59.923373 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:11:59.927211 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:11:59.927223 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:11:59.927238 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:11:59.927408 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:11:59.930178 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.930833 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.930840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.930843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.931000 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:11:59.931464 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:11:59.931467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:11:59.931469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:11:59.931820 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:11:59.932052 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 01:11:59.932267 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 01:11:59.932470 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":375,"threads_started":1}
I20260812 01:11:59.934074 31785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:00.022094 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 31562
I20260812 01:12:00.048763 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 30910
I20260812 01:12:00.050230 31022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:00.160470 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 30910
2026-08-12T01:12:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4355 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:12:00 dist-test-slave-21b9 krb5kdc[31841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:12:00 dist-test-slave-21b9 krb5kdc[31841](info): set up 2 sockets
Aug 12 01:12:00 dist-test-slave-21b9 krb5kdc[31841](info): commencing operation
krb5kdc: starting...
W20260812 01:12:01.462357 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:12:01 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:12:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.62.keytab.
Entry for principal oryx/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.62.keytab.
I20260812 01:12:01.749197 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:33689
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.62.keytab
--principal=oryx/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:33689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:02.095110 31862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:02.095456 31862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:02.095611 31862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:02.103044 31862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:12:02.103186 31862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:02.103286 31862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:02.103372 31862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:12:02.103434 31862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:12:02.113938 31862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:33689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.62.keytab
--principal=oryx/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:33689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:02.115449 31862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:02.116803 31862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:02.128636 31868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:02.128679 31869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:02.128723 31875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:02.129531 31862 server_base.cc:1034] running on GCE node
I20260812 01:12:02.130399 31862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:02.131933 31862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:02.133165 31862 hybrid_clock.cc:648] HybridClock initialized: now 1786497122133106 us; error 43 us; skew 500 ppm
I20260812 01:12:02.133546 31862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:02 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497122, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:02.136847 31862 init.cc:377] Logged in from keytab as oryx/127.20.151.62@KRBTEST.COM (short username oryx)
I20260812 01:12:02.138429 31862 webserver.cc:460] Webserver started at http://127.20.151.62:36349/ using document root <none> and password file <none>
I20260812 01:12:02.139189 31862 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:02.139358 31862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:02.139693 31862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:02.142879 31862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "cd3bef72f2d2436dbd3f6506788f0b68"
format_stamp: "Formatted at 2026-08-12 01:12:02 on dist-test-slave-21b9"
server_key: "c2cb7039518700db1bf4f52e38c1354d"
server_key_iv: "d1c6fd3bfbd74ee9b133af6325f1eb26"
server_key_version: "encryptionkey@0"
I20260812 01:12:02.143697 31862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "cd3bef72f2d2436dbd3f6506788f0b68"
format_stamp: "Formatted at 2026-08-12 01:12:02 on dist-test-slave-21b9"
server_key: "c2cb7039518700db1bf4f52e38c1354d"
server_key_iv: "d1c6fd3bfbd74ee9b133af6325f1eb26"
server_key_version: "encryptionkey@0"
I20260812 01:12:02.149190 31862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260812 01:12:02.152851 31885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:02.154362 31862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:12:02.154567 31862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "cd3bef72f2d2436dbd3f6506788f0b68"
format_stamp: "Formatted at 2026-08-12 01:12:02 on dist-test-slave-21b9"
server_key: "c2cb7039518700db1bf4f52e38c1354d"
server_key_iv: "d1c6fd3bfbd74ee9b133af6325f1eb26"
server_key_version: "encryptionkey@0"
I20260812 01:12:02.154809 31862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:02.174226 31862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:02.178766 31862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:02.179054 31862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:02.197908 31862 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:33689
I20260812 01:12:02.197948 31976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:33689 every 8 connection(s)
I20260812 01:12:02.199355 31862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:12:02.203229 31977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:02.207034 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 31862
I20260812 01:12:02.207302 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:12:02.208155 21084 external_mini_cluster.cc:1428] Setting key e8e15a137bad2af131dedf0412eb1f67
I20260812 01:12:02.217002 31977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68: Bootstrap starting.
I20260812 01:12:02.220950 31977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:02.222319 31977 log.cc:824] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68: Log is configured to *not* fsync() on all Append() calls
Aug 12 01:12:02 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:02.225513 31977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68: No bootstrap required, opened a new log
I20260812 01:12:02.232064 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:02.210328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.20.151.62:33689)
0812 01:12:02.211077 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:02.211088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:02.211150 (+ 62us) server_negotiation.cc:407] Connection header received
0812 01:12:02.212864 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:02.212887 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:02.213293 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:02.213633 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:02.215444 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:02.216794 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:02.218534 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:02.219334 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:02.219481 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:02.224270 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:02.224279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:02.224294 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:02.224344 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:02.227518 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:02.228338 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:02.228344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:02.228354 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:02.228573 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:02.228990 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:02.228993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:02.228996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:02.229631 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:02.229879 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:02.230460 (+ 581us) server_negotiation.cc:299] Negotiation successful
0812 01:12:02.230915 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":333,"threads_started":1}
I20260812 01:12:02.234735 31977 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } }
I20260812 01:12:02.235201 31977 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:02.235360 31977 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3bef72f2d2436dbd3f6506788f0b68, State: Initialized, Role: FOLLOWER
I20260812 01:12:02.235910 31977 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } }
I20260812 01:12:02.236122 31977 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:12:02.236250 31977 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:12:02.236395 31977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:02.238579 31977 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } }
I20260812 01:12:02.239099 31977 leader_election.cc:302] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd3bef72f2d2436dbd3f6506788f0b68; no voters:
I20260812 01:12:02.239800 31977 leader_election.cc:288] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:12:02.239940 31983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:02.240711 31983 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 1 LEADER]: Becoming Leader. State: Replica: cd3bef72f2d2436dbd3f6506788f0b68, State: Running, Role: LEADER
I20260812 01:12:02.241262 31983 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } }
I20260812 01:12:02.242008 31977 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:12:02.244266 31988 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd3bef72f2d2436dbd3f6506788f0b68. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } }
I20260812 01:12:02.244660 31988 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:02.244891 31986 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } }
I20260812 01:12:02.245254 31986 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:02.247498 31994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:12:02.252781 31994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:12:02.262766 31994 catalog_manager.cc:1347] Generated new cluster ID: 1c08e6e0a00e4060aacdd304bfe6627b
I20260812 01:12:02.262923 31994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:12:02.275556 31994 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:12:02.277192 31994 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:12:02.283319 31994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68: Generated new TSK 0
I20260812 01:12:02.284247 31994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab.
Entry for principal oryx/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab.
I20260812 01:12:02.358697 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:33689
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab
--principal=oryx/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:02.722695 32022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:02.723029 32022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:02.723184 32022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:02.731325 32022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:02.731482 32022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:02.731665 32022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:12:02.742019 32022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab
--principal=oryx/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:02.743573 32022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:02.745082 32022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:02.759420 32032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:02.759457 32030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:02.759609 32029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:02.760183 32022 server_base.cc:1034] running on GCE node
I20260812 01:12:02.760886 32022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:02.761909 32022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:02.763154 32022 hybrid_clock.cc:648] HybridClock initialized: now 1786497122763087 us; error 48 us; skew 500 ppm
I20260812 01:12:02.763569 32022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:02 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497122, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:02.770843 32022 init.cc:377] Logged in from keytab as oryx/127.20.151.1@KRBTEST.COM (short username oryx)
I20260812 01:12:02.772420 32022 webserver.cc:460] Webserver started at http://127.20.151.1:38069/ using document root <none> and password file <none>
I20260812 01:12:02.773124 32022 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:02.773253 32022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:02.773543 32022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:02.776588 32022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "dd7fce201fee4ca3b83deab3cb7ac345"
format_stamp: "Formatted at 2026-08-12 01:12:02 on dist-test-slave-21b9"
server_key: "806af2a5293b48ddd0ba0a452f021cfd"
server_key_iv: "e07a02bc8a9f2133dfb273670b13291c"
server_key_version: "encryptionkey@0"
I20260812 01:12:02.777385 32022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "dd7fce201fee4ca3b83deab3cb7ac345"
format_stamp: "Formatted at 2026-08-12 01:12:02 on dist-test-slave-21b9"
server_key: "806af2a5293b48ddd0ba0a452f021cfd"
server_key_iv: "e07a02bc8a9f2133dfb273670b13291c"
server_key_version: "encryptionkey@0"
I20260812 01:12:02.783039 32022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:12:02.786788 32045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:02.788300 32022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:12:02.788471 32022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "dd7fce201fee4ca3b83deab3cb7ac345"
format_stamp: "Formatted at 2026-08-12 01:12:02 on dist-test-slave-21b9"
server_key: "806af2a5293b48ddd0ba0a452f021cfd"
server_key_iv: "e07a02bc8a9f2133dfb273670b13291c"
server_key_version: "encryptionkey@0"
I20260812 01:12:02.788689 32022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:02.837842 32022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:02.842350 32022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:02.842689 32022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:02.843779 32022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:02.845182 32022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:02.845263 32022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:02.845393 32022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:02.845448 32022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:02.877686 32022 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:33281
I20260812 01:12:02.877727 32223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:33281 every 8 connection(s)
I20260812 01:12:02.878926 32022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:12:02.888649 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 32022
I20260812 01:12:02.888865 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:12:02.889745 21084 external_mini_cluster.cc:1428] Setting key aa40d88f031162f7fa90206f052836d7
Aug 12 01:12:02 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497122, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:02.900887 32226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:02.881559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:54351 (local address 127.20.151.62:33689)
0812 01:12:02.882034 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:02.882038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:02.882856 (+ 818us) server_negotiation.cc:407] Connection header received
0812 01:12:02.884928 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:02.884933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:02.885031 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:02.885129 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:02.886811 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:02.887632 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:02.889842 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:02.890267 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:02.890373 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:02.894215 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:02.894222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:02.894227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:02.894268 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:02.897391 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:02.898148 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:02.898153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:02.898157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:02.898303 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:02.898823 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:02.898827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:02.898830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:02.899170 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:02.899331 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:02.900378 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0812 01:12:02.900577 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":263,"threads_started":1}
I20260812 01:12:02.902160 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:02.882002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.1:54351)
0812 01:12:02.882699 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:02.882739 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:02.884652 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:02.885429 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:02.885438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:02.886012 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:02.886624 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:02.886654 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:02.887801 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:02.887805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:02.889675 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:02.889690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:02.890529 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:02.890538 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:02.890754 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:02.891530 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:02.891559 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:02.893984 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:02.897551 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:02.897559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:02.897574 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:02.898033 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:02.898481 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:02.898485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:02.898488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:02.898701 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:02.899415 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:02.899423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:02.899902 (+ 479us) client_negotiation.cc:769] Sending connection context
0812 01:12:02.900220 (+ 318us) client_negotiation.cc:240] Negotiation successful
0812 01:12:02.900790 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":259,"threads_started":1}
I20260812 01:12:02.903842 32224 heartbeater.cc:347] Connected to a master server at 127.20.151.62:33689
I20260812 01:12:02.904147 32224 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:02.904884 32224 heartbeater.cc:511] Master 127.20.151.62:33689 requested a full tablet report, sending...
I20260812 01:12:02.907001 31917 ts_manager.cc:194] Registered new tserver with Master: dd7fce201fee4ca3b83deab3cb7ac345 (127.20.151.1:33281)
I20260812 01:12:02.908520 31917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:54351
WARNING: no policy specified for oryx/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab.
Entry for principal oryx/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab.
I20260812 01:12:02.952262 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:33689
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab
--principal=oryx/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:03.309849 32232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:03.310321 32232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:03.310515 32232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:03.318156 32232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:03.318282 32232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:03.318449 32232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:12:03.329089 32232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab
--principal=oryx/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:03.330449 32232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:03.331905 32232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:03.345547 32247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:03.345444 32248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:03.346058 32251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:03.346318 32232 server_base.cc:1034] running on GCE node
I20260812 01:12:03.347003 32232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:03.348068 32232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:03.349326 32232 hybrid_clock.cc:648] HybridClock initialized: now 1786497123349271 us; error 41 us; skew 500 ppm
I20260812 01:12:03.349700 32232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:03 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497123, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:03.353129 32232 init.cc:377] Logged in from keytab as oryx/127.20.151.2@KRBTEST.COM (short username oryx)
I20260812 01:12:03.354820 32232 webserver.cc:460] Webserver started at http://127.20.151.2:34561/ using document root <none> and password file <none>
I20260812 01:12:03.355603 32232 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:03.355727 32232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:03.356040 32232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:03.359221 32232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "1955a48e1490429484d27bf69dd64ab5"
format_stamp: "Formatted at 2026-08-12 01:12:03 on dist-test-slave-21b9"
server_key: "95bac1fe70581a2ff690e9df74a8fe2e"
server_key_iv: "0378ed05156cbd3f0b2bf0bd28151ac4"
server_key_version: "encryptionkey@0"
I20260812 01:12:03.360060 32232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "1955a48e1490429484d27bf69dd64ab5"
format_stamp: "Formatted at 2026-08-12 01:12:03 on dist-test-slave-21b9"
server_key: "95bac1fe70581a2ff690e9df74a8fe2e"
server_key_iv: "0378ed05156cbd3f0b2bf0bd28151ac4"
server_key_version: "encryptionkey@0"
I20260812 01:12:03.365437 32232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:12:03.369129 32263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:03.370539 32232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:12:03.370707 32232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "1955a48e1490429484d27bf69dd64ab5"
format_stamp: "Formatted at 2026-08-12 01:12:03 on dist-test-slave-21b9"
server_key: "95bac1fe70581a2ff690e9df74a8fe2e"
server_key_iv: "0378ed05156cbd3f0b2bf0bd28151ac4"
server_key_version: "encryptionkey@0"
I20260812 01:12:03.370904 32232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:03.396011 32232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:03.400131 32232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:03.400411 32232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:03.401686 32232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:03.403167 32232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:03.403252 32232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:03.403394 32232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:03.403450 32232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:03.435316 32232 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:34343
I20260812 01:12:03.435376 32462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:34343 every 8 connection(s)
I20260812 01:12:03.436648 32232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:12:03.441026 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 32232
I20260812 01:12:03.441224 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:12:03.441823 21084 external_mini_cluster.cc:1428] Setting key bf90ebd45a723005dcbac3f55e82d404
Aug 12 01:12:03 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497123, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:03.460125 32467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:03.439241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:47763 (local address 127.20.151.62:33689)
0812 01:12:03.439711 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:03.439717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:03.440889 (+ 1172us) server_negotiation.cc:407] Connection header received
0812 01:12:03.443519 (+ 2630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:03.443525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:03.443596 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:03.443686 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:03.445752 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:03.446479 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:03.448597 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:03.449019 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:03.449117 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:03.453319 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:03.453324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:03.453328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:03.453365 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:03.456024 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:03.456796 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:03.456801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:03.456804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:03.456966 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:03.457370 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:03.457374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:03.457377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:03.457745 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:03.457885 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:03.459475 (+ 1590us) server_negotiation.cc:299] Negotiation successful
0812 01:12:03.459759 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":193,"threads_started":1}
I20260812 01:12:03.460667 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:03.439867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.2:47763)
0812 01:12:03.440692 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:03.440752 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:03.443138 (+ 2386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:03.443962 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:03.443971 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:03.444700 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:03.445547 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:03.445572 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:03.446654 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:03.446660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:03.448460 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:03.448475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:03.449265 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:03.449271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:03.449545 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:03.450268 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:03.450297 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:03.453041 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:03.456188 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:03.456198 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:03.456217 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:03.456663 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:03.457060 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:03.457063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:03.457065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:03.457259 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:03.457891 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:03.457899 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:03.458358 (+ 459us) client_negotiation.cc:769] Sending connection context
0812 01:12:03.458688 (+ 330us) client_negotiation.cc:240] Negotiation successful
0812 01:12:03.459249 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":313,"threads_started":1}
I20260812 01:12:03.462112 32465 heartbeater.cc:347] Connected to a master server at 127.20.151.62:33689
I20260812 01:12:03.462469 32465 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:03.463405 32465 heartbeater.cc:511] Master 127.20.151.62:33689 requested a full tablet report, sending...
I20260812 01:12:03.465202 31917 ts_manager.cc:194] Registered new tserver with Master: 1955a48e1490429484d27bf69dd64ab5 (127.20.151.2:34343)
I20260812 01:12:03.466058 31917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:47763
WARNING: no policy specified for oryx/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab.
Entry for principal oryx/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab.
I20260812 01:12:03.507495 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:33689
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab
--principal=oryx/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:03.848055 32481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:03.848359 32481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:03.848464 32481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:03.855746 32481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:03.855852 32481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:03.855973 32481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:12:03.865805 32481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab
--principal=oryx/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:03.867138 32481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:03.868587 32481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:03.881997 32490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:03.882026 32492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:03.883070 32481 server_base.cc:1034] running on GCE node
W20260812 01:12:03.883260 32487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:03.883929 32481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:03.884917 32481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:03.886142 32481 hybrid_clock.cc:648] HybridClock initialized: now 1786497123886076 us; error 48 us; skew 500 ppm
I20260812 01:12:03.886467 32481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:03 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497123, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:03.889623 32481 init.cc:377] Logged in from keytab as oryx/127.20.151.3@KRBTEST.COM (short username oryx)
I20260812 01:12:03.891122 32481 webserver.cc:460] Webserver started at http://127.20.151.3:37547/ using document root <none> and password file <none>
I20260812 01:12:03.891855 32481 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:03.891987 32481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:03.892297 32481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:03.895323 32481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "eec8159f12d9469aa0168790d61e4bda"
format_stamp: "Formatted at 2026-08-12 01:12:03 on dist-test-slave-21b9"
server_key: "bbe27d84fb3321ea634b016d5ce73ca8"
server_key_iv: "b18bc0ce78d8e3e7dac32d96cbf8bbd2"
server_key_version: "encryptionkey@0"
I20260812 01:12:03.896098 32481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "eec8159f12d9469aa0168790d61e4bda"
format_stamp: "Formatted at 2026-08-12 01:12:03 on dist-test-slave-21b9"
server_key: "bbe27d84fb3321ea634b016d5ce73ca8"
server_key_iv: "b18bc0ce78d8e3e7dac32d96cbf8bbd2"
server_key_version: "encryptionkey@0"
I20260812 01:12:03.901197 32481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:12:03.904716 32501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:03.906024 32481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:12:03.906172 32481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "eec8159f12d9469aa0168790d61e4bda"
format_stamp: "Formatted at 2026-08-12 01:12:03 on dist-test-slave-21b9"
server_key: "bbe27d84fb3321ea634b016d5ce73ca8"
server_key_iv: "b18bc0ce78d8e3e7dac32d96cbf8bbd2"
server_key_version: "encryptionkey@0"
I20260812 01:12:03.906354 32481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:03.911547 32224 heartbeater.cc:503] Master 127.20.151.62:33689 was elected leader, sending a full tablet report...
I20260812 01:12:03.935001 32481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:03.939060 32481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:03.939368 32481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:03.940347 32481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:03.941769 32481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:03.941841 32481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:03.941941 32481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:03.941991 32481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:03.973941 32481 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:45441
I20260812 01:12:03.974022 32691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:45441 every 8 connection(s)
I20260812 01:12:03.975306 32481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:12:03.975831 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 32481
I20260812 01:12:03.976002 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:12:03.976583 21084 external_mini_cluster.cc:1428] Setting key 91c857aed1190bc049612b4776cd1682
Aug 12 01:12:03 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497123, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:03.995457 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:03.978042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:42593 (local address 127.20.151.62:33689)
0812 01:12:03.978496 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:03.978502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:03.979623 (+ 1121us) server_negotiation.cc:407] Connection header received
0812 01:12:03.981468 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:03.981472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:03.981546 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:03.981635 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:03.983030 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:03.983879 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:03.985372 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:03.985756 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:03.985845 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:03.989246 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:03.989250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:03.989254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:03.989287 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:03.992514 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:03.993311 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:03.993316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:03.993319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:03.993489 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:03.993911 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:03.993914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:03.993916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:03.994191 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:03.994311 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:03.994958 (+ 647us) server_negotiation.cc:299] Negotiation successful
0812 01:12:03.995136 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":213,"threads_started":1}
I20260812 01:12:03.996472 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:03.978658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.3:42593)
0812 01:12:03.979488 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:03.979533 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:03.981232 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:03.981823 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:03.981832 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:03.982332 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:03.982889 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:03.982907 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:03.984026 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:03.984030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:03.985256 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:03.985269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:03.985926 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:03.985931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:03.986146 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:03.986814 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:03.986838 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:03.989036 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:03.992738 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:03.992745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:03.992760 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:03.993202 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:03.993619 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:03.993622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:03.993625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:03.993825 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:03.994292 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:03.994298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:03.994641 (+ 343us) client_negotiation.cc:769] Sending connection context
0812 01:12:03.994875 (+ 234us) client_negotiation.cc:240] Negotiation successful
0812 01:12:03.995317 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":350,"threads_started":1}
I20260812 01:12:03.997895 32692 heartbeater.cc:347] Connected to a master server at 127.20.151.62:33689
I20260812 01:12:03.998196 32692 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:03.998920 32692 heartbeater.cc:511] Master 127.20.151.62:33689 requested a full tablet report, sending...
I20260812 01:12:04.000339 31917 ts_manager.cc:194] Registered new tserver with Master: eec8159f12d9469aa0168790d61e4bda (127.20.151.3:45441)
I20260812 01:12:04.001140 31917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:42593
I20260812 01:12:04.005054 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.151.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.61.keytab.
Entry for principal oryx/127.20.151.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.61.keytab.
I20260812 01:12:04.072643 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.61:40015
--webserver_interface=127.20.151.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.61.keytab
--principal=oryx/127.20.151.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:33689,127.20.151.61:40015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:12:04.469430 32465 heartbeater.cc:503] Master 127.20.151.62:33689 was elected leader, sending a full tablet report...
W20260812 01:12:04.500893 32706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:04.501194 32706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:04.501294 32706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:04.508862 32706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:12:04.508975 32706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:04.509032 32706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:04.509091 32706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:12:04.509135 32706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:12:04.519369 32706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:33689,127.20.151.61:40015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.61.keytab
--principal=oryx/127.20.151.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.61:40015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.151.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
W20260812 01:12:04.519731 32706 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.151.62:33689,127.20.151.61:40015'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 01:12:04.520768 32706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:04.522048 32706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:04.533981 32724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:04.534020 32719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:04.534072 32718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:04.537025 32706 server_base.cc:1034] running on GCE node
I20260812 01:12:04.537914 32706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:04.539446 32706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:04.540699 32706 hybrid_clock.cc:648] HybridClock initialized: now 1786497124540627 us; error 39 us; skew 500 ppm
I20260812 01:12:04.541030 32706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:04 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497124, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:04.544226 32706 init.cc:377] Logged in from keytab as oryx/127.20.151.61@KRBTEST.COM (short username oryx)
I20260812 01:12:04.545785 32706 webserver.cc:460] Webserver started at http://127.20.151.61:36403/ using document root <none> and password file <none>
I20260812 01:12:04.546463 32706 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:04.546557 32706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:04.546859 32706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:04.549928 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data/instance:
uuid: "9d65031b3e3745329fdb09dcf3882d61"
format_stamp: "Formatted at 2026-08-12 01:12:04 on dist-test-slave-21b9"
server_key: "9961d2b669753d0601ce75fd8ddfbbc3"
server_key_iv: "c5f6da978a66d436a29d73d22b808b30"
server_key_version: "encryptionkey@0"
I20260812 01:12:04.550709 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/wal/instance:
uuid: "9d65031b3e3745329fdb09dcf3882d61"
format_stamp: "Formatted at 2026-08-12 01:12:04 on dist-test-slave-21b9"
server_key: "9961d2b669753d0601ce75fd8ddfbbc3"
server_key_iv: "c5f6da978a66d436a29d73d22b808b30"
server_key_version: "encryptionkey@0"
I20260812 01:12:04.556082 32706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 01:12:04.559847 32736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:04.561347 32706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:12:04.561510 32706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/wal
uuid: "9d65031b3e3745329fdb09dcf3882d61"
format_stamp: "Formatted at 2026-08-12 01:12:04 on dist-test-slave-21b9"
server_key: "9961d2b669753d0601ce75fd8ddfbbc3"
server_key_iv: "c5f6da978a66d436a29d73d22b808b30"
server_key_version: "encryptionkey@0"
I20260812 01:12:04.561705 32706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:04.578830 32706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:04.583151 32706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:04.583446 32706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:04.601377 32706 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.61:40015
I20260812 01:12:04.601383 348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.61:40015 every 8 connection(s)
I20260812 01:12:04.602767 32706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
I20260812 01:12:04.606508 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:04.608106 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 32706
I20260812 01:12:04.616493 350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:12:04 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.151.61@KRBTEST.COM
Aug 12 01:12:04 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497124, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.61@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:04.633131 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.610590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.20.151.61:40015)
0812 01:12:04.611463 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:04.611477 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:04.611527 (+ 50us) server_negotiation.cc:407] Connection header received
0812 01:12:04.613158 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:04.613182 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:04.613595 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:04.613935 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:04.615806 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.617171 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.618933 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.620123 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.620349 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.625366 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:04.625374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:04.625387 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:04.625427 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:04.628677 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.629471 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.629480 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.629490 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.629652 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.630065 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.630070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.630074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.630646 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:04.630915 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:04.631409 (+ 494us) server_negotiation.cc:299] Negotiation successful
0812 01:12:04.631843 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":390,"threads_started":1}
I20260812 01:12:04.635556 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.619543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40986 (local address 127.20.151.62:33689)
0812 01:12:04.620018 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:04.620024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:04.620704 (+ 680us) server_negotiation.cc:407] Connection header received
0812 01:12:04.620897 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:04.620904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:04.621008 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:04.621118 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:04.622642 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.623462 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.625145 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.625564 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.625680 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.629270 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:04.629278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:04.629282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:04.629329 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:04.632373 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.633266 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.633272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.633274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.633439 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.633865 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.633869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.633871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.634202 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:04.634316 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:04.634907 (+ 591us) server_negotiation.cc:299] Negotiation successful
0812 01:12:04.635140 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260812 01:12:04.635629 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.619683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:40986)
0812 01:12:04.620537 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:04.620591 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:04.620770 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:04.621204 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:04.621211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:04.622112 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:04.622459 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.622473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.623612 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.623617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.624987 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.625004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.625765 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.625772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.625960 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.626513 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:04.626532 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:04.629041 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:04.632518 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.632529 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.632535 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.633106 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.633536 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.633540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.633543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.633729 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.634339 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:04.634346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:04.634547 (+ 201us) client_negotiation.cc:769] Sending connection context
0812 01:12:04.634787 (+ 240us) client_negotiation.cc:240] Negotiation successful
0812 01:12:04.635136 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":416,"threads_started":1}
I20260812 01:12:04.637665 350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:12:04 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497124, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.61@KRBTEST.COM for oryx/127.20.151.61@KRBTEST.COM
I20260812 01:12:04.650126 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.638274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:40015 (local address 127.0.0.1:57876)
0812 01:12:04.638497 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:04.638516 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:04.638601 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:04.638931 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:04.638934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:04.639302 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:04.639442 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.639450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.640427 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.640431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.641296 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.641307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.642023 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.642028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.642166 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.642781 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:04.642798 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:04.644883 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:04.647840 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.647844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.647847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.648319 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.648632 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.648636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.648638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.648769 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.649300 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:04.649303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:04.649433 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 01:12:04.649543 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 01:12:04.649775 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 01:12:04.650365 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.638566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.20.151.61:40015)
0812 01:12:04.638715 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:04.638719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:04.638733 (+ 14us) server_negotiation.cc:407] Connection header received
0812 01:12:04.638778 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:04.638781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:04.638834 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:04.638909 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:04.639584 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.640301 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.641426 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.641854 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.641952 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.645060 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:04.645065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:04.645068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:04.645098 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:04.647720 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.648414 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.648420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.648423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.648546 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.648862 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.648865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.648867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.649197 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:04.649305 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:04.649622 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 01:12:04.649963 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":48}
I20260812 01:12:04.656077 350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: Bootstrap starting.
I20260812 01:12:04.659358 350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:04.660487 350 log.cc:824] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:04.663196 350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: No bootstrap required, opened a new log
I20260812 01:12:04.670547 350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } }
I20260812 01:12:04.670893 350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:04.671000 350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d65031b3e3745329fdb09dcf3882d61, State: Initialized, Role: FOLLOWER
I20260812 01:12:04.671609 350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } }
I20260812 01:12:04.672538 364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } } }
I20260812 01:12:04.672860 364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:12:04.673529 350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:12:04.674899 364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:12:04.675230 364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } }
I20260812 01:12:04.676925 364 leader_election.cc:288] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3bef72f2d2436dbd3f6506788f0b68 (127.20.151.62:33689)
I20260812 01:12:04.678097 31934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d65031b3e3745329fdb09dcf3882d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" is_pre_election: true
I20260812 01:12:04.679540 32739 leader_election.cc:302] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 9d65031b3e3745329fdb09dcf3882d61; no voters: cd3bef72f2d2436dbd3f6506788f0b68
I20260812 01:12:04.680184 364 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 01:12:04.685922 32706 master_runner.cc:428] Detected that this master 9d65031b3e3745329fdb09dcf3882d61 is joining an existing cluster
I20260812 01:12:04.686045 32706 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.151.61:40015
W20260812 01:12:04.686125 381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:12:04.686262 381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:12:04.708655 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.690196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.20.151.62:33689)
0812 01:12:04.690379 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:04.690385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:04.691024 (+ 639us) server_negotiation.cc:407] Connection header received
0812 01:12:04.692705 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:04.692711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:04.692814 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:04.692891 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:04.694627 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.695843 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.698125 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.698847 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.698948 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.702603 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:04.702609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:04.702612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:04.702647 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:04.705719 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.706338 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.706343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.706347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.706498 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.706922 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.706927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.706931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.707344 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:04.707481 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:04.708015 (+ 534us) server_negotiation.cc:299] Negotiation successful
0812 01:12:04.708252 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:12:04.708712 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.690240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:40996)
0812 01:12:04.690931 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:04.690949 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:04.691136 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:04.693138 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:04.693144 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:04.693977 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:04.694432 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.694447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.696047 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.696053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.697928 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.697947 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.699308 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.699316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.699525 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.700325 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:04.700351 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:04.702417 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:04.705848 (+ 3431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.705853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.705857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.706224 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.706618 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.706623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.706627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.706807 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.707532 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:04.707535 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:04.707707 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 01:12:04.707873 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 01:12:04.708224 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":371,"threads_started":1}
I20260812 01:12:04.716446 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.691072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:40015 (local address 127.0.0.1:57890)
0812 01:12:04.701879 (+ 10807us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:04.701896 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:04.702023 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:04.702377 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:04.702383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:04.703079 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:04.703430 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.703444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.705089 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.705095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.706362 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.706377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.707335 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.707345 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.707509 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.708140 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:04.708163 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:04.709931 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:04.713640 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.713644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.713647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.714153 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.714513 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.714516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.714519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.714667 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.715425 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:04.715428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:04.715558 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 01:12:04.715680 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 01:12:04.715983 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10685,"thread_start_us":10469,"threads_started":1}
I20260812 01:12:04.717372 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.691393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.20.151.61:40015)
0812 01:12:04.692539 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:04.692546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:04.701989 (+ 9443us) server_negotiation.cc:407] Connection header received
0812 01:12:04.702125 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:04.702129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:04.702208 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:04.702309 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:04.703592 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.704884 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.706499 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.707078 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.707198 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.710091 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:04.710098 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:04.710103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:04.710138 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:04.713501 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.714268 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.714272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.714275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.714412 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.714769 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.714773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.714776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.715189 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:04.715373 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:04.716765 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0812 01:12:04.716999 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980}
WARNING: no policy specified for oryx/127.20.151.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.60@KRBTEST.COM" created.
Aug 12 01:12:04 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497122, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.62@KRBTEST.COM for oryx/127.20.151.61@KRBTEST.COM
I20260812 01:12:04.745153 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.716355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.20.151.61:40015)
0812 01:12:04.717664 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:04.717671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:04.729356 (+ 11685us) server_negotiation.cc:407] Connection header received
0812 01:12:04.729753 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:04.729759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:04.729863 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:04.729978 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:04.731786 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.732644 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.734576 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.735488 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.735625 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.739149 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:04.739155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:04.739158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:04.739191 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:04.742177 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.742952 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.742958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.742963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.743112 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.743574 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.743578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.743581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.743894 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:04.744038 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:04.744530 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 01:12:04.744757 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202,"thread_start_us":11106,"threads_started":1}
I20260812 01:12:04.745185 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.716424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:40015 (local address 127.0.0.1:57892)
0812 01:12:04.729173 (+ 12749us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:04.729228 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:04.729475 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:04.730256 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:04.730266 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:04.731187 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:04.731583 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.731600 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.732798 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.732804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.734414 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.734431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.735884 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.735891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.736085 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.736736 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:04.736766 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:04.738954 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:04.742325 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.742339 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.742348 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.742846 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.743223 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.743229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.743233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.743448 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.744025 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:04.744034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:04.744226 (+ 192us) client_negotiation.cc:769] Sending connection context
0812 01:12:04.744383 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 01:12:04.744748 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12427,"thread_start_us":13262,"threads_started":1}
I20260812 01:12:04.747406 31917 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.151.61:40015: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61"
member_type: NON_VOTER
last_known_addr {
host: "127.20.151.61"
port: 40015
}
attrs {
promote: true
}
}
dest_uuid: "cd3bef72f2d2436dbd3f6506788f0b68"
cas_config_opid_index: -1
I20260812 01:12:04.748517 31917 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } }
I20260812 01:12:04.751878 409 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9d65031b3e3745329fdb09dcf3882d61 (127.20.151.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } }
I20260812 01:12:04.753082 312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:04.756975 312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 FOLLOWER]: Refusing update from remote peer cd3bef72f2d2436dbd3f6506788f0b68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 01:12:04.756536 409 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.151.61:40015
I20260812 01:12:04.758599 32706 master.cc:537] Master@127.20.151.61:40015 shutting down...
I20260812 01:12:04.759512 409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:04.760218 410 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } }
I20260812 01:12:04.760686 410 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:04.773074 32706 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:04.773496 32706 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:04.773679 32706 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: stopping tablet replica
W20260812 01:12:04.775453 31887 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260812 01:12:04.778090 31887 consensus_peers.cc:487] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 -> Peer 9d65031b3e3745329fdb09dcf3882d61 (127.20.151.61:40015): Couldn't send request to peer 9d65031b3e3745329fdb09dcf3882d61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 01:12:04.791302 32706 master.cc:552] Master@127.20.151.61:40015 shutdown complete.
I20260812 01:12:04.791476 32706 master_runner.cc:305] Clearing existing system tablet
I20260812 01:12:04.792003 32706 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:04.792107 32706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.60.keytab.
Entry for principal oryx/127.20.151.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.60.keytab.
W20260812 01:12:04.794200 21084 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.20.151.60: real 0.105s user 0.006s sys 0.019s
I20260812 01:12:04.795199 32706 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 01:12:04.795403 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.60:34219
--webserver_interface=127.20.151.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.60.keytab
--principal=oryx/127.20.151.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:33689,127.20.151.61:40015,127.20.151.60:34219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:12:04.796967 421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:04.797348 32706 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:04.797667 32706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 01:12:04.797755 32706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/wal
uuid: "9d65031b3e3745329fdb09dcf3882d61"
format_stamp: "Formatted at 2026-08-12 01:12:04 on dist-test-slave-21b9"
server_key: "9961d2b669753d0601ce75fd8ddfbbc3"
server_key_iv: "c5f6da978a66d436a29d73d22b808b30"
server_key_version: "encryptionkey@0"
I20260812 01:12:04.802933 32706 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:04.810259 32706 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 01:12:04.810474 32706 log.cc:1196] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 01:12:04.810981 32706 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: Deleting consensus metadata
I20260812 01:12:04.811302 32706 master_runner.cc:315] Copying system tablet from 127.20.151.62:33689
I20260812 01:12:04.817005 32706 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.62:33689
I20260812 01:12:04.838694 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.817959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.20.151.62:33689)
0812 01:12:04.818165 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:04.818171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:04.823932 (+ 5761us) server_negotiation.cc:407] Connection header received
0812 01:12:04.824141 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:04.824148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:04.824289 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:04.824417 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:04.825568 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.826771 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.828696 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.829382 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.829505 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.831934 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:04.831942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:04.831947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:04.831990 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:04.835469 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.836323 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.836329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.836332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.836511 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:04.837008 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:04.837014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:04.837017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:04.837419 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:04.837598 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:04.838049 (+ 451us) server_negotiation.cc:299] Negotiation successful
0812 01:12:04.838299 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 01:12:04.838716 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:04.817919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:41010)
0812 01:12:04.823763 (+ 5844us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:04.823792 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:04.823977 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:04.824471 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:04.824475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:04.825046 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:04.825385 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.825399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.826924 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.826930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.828534 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:04.828560 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:04.829548 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:04.829555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:04.829754 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:04.830464 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:04.830489 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:04.831708 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:04.835630 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.835636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.835641 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.836182 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.836666 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:04.836672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:04.836680 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:04.836869 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:04.837559 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:04.837564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:04.837753 (+ 189us) client_negotiation.cc:769] Sending connection context
0812 01:12:04.837915 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 01:12:04.838266 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5664,"thread_start_us":5518,"threads_started":1}
I20260812 01:12:04.839922 31951 tablet_copy_service.cc:140] P cd3bef72f2d2436dbd3f6506788f0b68: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9d65031b3e3745329fdb09dcf3882d61 ({username='oryx', principal='oryx/127.20.151.61@KRBTEST.COM'} at 127.0.0.1:41010)
I20260812 01:12:04.840215 31951 tablet_copy_service.cc:161] P cd3bef72f2d2436dbd3f6506788f0b68: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9d65031b3e3745329fdb09dcf3882d61 at {username='oryx', principal='oryx/127.20.151.61@KRBTEST.COM'} at 127.0.0.1:41010: session id = 9d65031b3e3745329fdb09dcf3882d61-00000000000000000000000000000000
I20260812 01:12:04.844874 31951 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 01:12:04.847376 32706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 01:12:04.854375 32706 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: tablet copy: Starting download of 0 data blocks...
I20260812 01:12:04.854704 32706 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: tablet copy: Starting download of 1 WAL segments...
I20260812 01:12:04.857542 32706 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 01:12:04.862445 31951 tablet_copy_service.cc:342] P cd3bef72f2d2436dbd3f6506788f0b68: Request end of tablet copy session 9d65031b3e3745329fdb09dcf3882d61-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.151.61@KRBTEST.COM'} at 127.0.0.1:41010
I20260812 01:12:04.862655 31951 tablet_copy_service.cc:433] P cd3bef72f2d2436dbd3f6506788f0b68: ending tablet copy session 9d65031b3e3745329fdb09dcf3882d61-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9d65031b3e3745329fdb09dcf3882d61
I20260812 01:12:04.880836 32706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:04.887619 441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:04.889827 444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:04.894248 446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:04.895635 32706 server_base.cc:1034] running on GCE node
I20260812 01:12:04.896051 32706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:04.899541 32706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:04.900867 32706 hybrid_clock.cc:648] HybridClock initialized: now 1786497124900749 us; error 85 us; skew 500 ppm
I20260812 01:12:04.901147 32706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:04 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497124, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:04.903468 32706 init.cc:377] Logged in from keytab as oryx/127.20.151.61@KRBTEST.COM (short username oryx)
I20260812 01:12:04.904839 32706 webserver.cc:460] Webserver started at http://127.20.151.61:44847/ using document root <none> and password file <none>
I20260812 01:12:04.905215 32706 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:04.905298 32706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:04.908571 32706 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 01:12:04.910796 457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:04.911440 32706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 01:12:04.911588 32706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/wal
uuid: "9d65031b3e3745329fdb09dcf3882d61"
format_stamp: "Formatted at 2026-08-12 01:12:04 on dist-test-slave-21b9"
server_key: "9961d2b669753d0601ce75fd8ddfbbc3"
server_key_iv: "c5f6da978a66d436a29d73d22b808b30"
server_key_version: "encryptionkey@0"
I20260812 01:12:04.911770 32706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:04.934805 32706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:04.939258 32706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:04.973022 32706 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.61:40015
I20260812 01:12:04.973083 539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.61:40015 every 8 connection(s)
I20260812 01:12:04.974707 32706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
I20260812 01:12:04.985282 542 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:12:04.989125 542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: Bootstrap starting.
I20260812 01:12:05.003782 32692 heartbeater.cc:503] Master 127.20.151.62:33689 was elected leader, sending a full tablet report...
I20260812 01:12:05.007568 542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:12:05.008188 542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: Bootstrap complete.
I20260812 01:12:05.009467 542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } }
I20260812 01:12:05.009755 542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d65031b3e3745329fdb09dcf3882d61, State: Initialized, Role: LEARNER
I20260812 01:12:05.010010 542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } }
I20260812 01:12:05.011011 550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } }
I20260812 01:12:05.011300 542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:12:05.011718 550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: This master's current role is: LEARNER
I20260812 01:12:05.035362 563 catalog_manager.cc:1259] Loaded cluster ID: 1c08e6e0a00e4060aacdd304bfe6627b
I20260812 01:12:05.035619 563 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: loading cluster ID for follower catalog manager: success
I20260812 01:12:05.039876 563 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: acquiring CA information for follower catalog manager: success
I20260812 01:12:05.043316 563 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:12:05.249456 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.241554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.20.151.61:40015)
0812 01:12:05.242426 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:05.242431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:05.242448 (+ 17us) server_negotiation.cc:407] Connection header received
0812 01:12:05.242536 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:05.242541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:05.242635 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:05.242738 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:05.243308 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.244611 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.247094 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.248204 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.248320 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.248591 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:05.248925 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 01:12:05.249034 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":576,"threads_started":1}
I20260812 01:12:05.249446 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.241228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:40015 (local address 127.0.0.1:57906)
0812 01:12:05.241463 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:05.241487 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:05.241638 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:05.242815 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:05.242821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:05.242855 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:05.243090 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.243113 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.244771 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.244777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.246937 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.246956 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.248387 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.248394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.248725 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.248794 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 01:12:05.248907 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 01:12:05.249059 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 01:12:05.250972 501 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260812 01:12:05.251752 550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd3bef72f2d2436dbd3f6506788f0b68. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } }
I20260812 01:12:05.252028 550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: This master's current role is: LEARNER
W20260812 01:12:05.292953 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:05.293406 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:05.293593 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:05.302414 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:12:05.302558 424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:05.302644 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:05.302719 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:12:05.302781 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:12:05.315686 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:33689,127.20.151.61:40015,127.20.151.60:34219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.60.keytab
--principal=oryx/127.20.151.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.60:34219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.151.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:05.317152 424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:05.318656 424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:05.334415 583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:05.334529 579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:05.335216 576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:05.335793 424 server_base.cc:1034] running on GCE node
I20260812 01:12:05.336874 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:05.338728 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:05.340005 424 hybrid_clock.cc:648] HybridClock initialized: now 1786497125339889 us; error 64 us; skew 500 ppm
I20260812 01:12:05.340420 424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:05 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497125, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:05.345302 424 init.cc:377] Logged in from keytab as oryx/127.20.151.60@KRBTEST.COM (short username oryx)
I20260812 01:12:05.347146 424 webserver.cc:460] Webserver started at http://127.20.151.60:44437/ using document root <none> and password file <none>
I20260812 01:12:05.348001 424 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:05.348202 424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:05.348493 424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:05.352241 424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data/instance:
uuid: "a749f9fa01ae42248b34e414f8c225c0"
format_stamp: "Formatted at 2026-08-12 01:12:05 on dist-test-slave-21b9"
server_key: "f8b31792fe478fe94371e643c5dad8dc"
server_key_iv: "1f0441cd41f3ed56ec80a4f3d37f11e2"
server_key_version: "encryptionkey@0"
I20260812 01:12:05.353106 424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/wal/instance:
uuid: "a749f9fa01ae42248b34e414f8c225c0"
format_stamp: "Formatted at 2026-08-12 01:12:05 on dist-test-slave-21b9"
server_key: "f8b31792fe478fe94371e643c5dad8dc"
server_key_iv: "1f0441cd41f3ed56ec80a4f3d37f11e2"
server_key_version: "encryptionkey@0"
I20260812 01:12:05.359784 424 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 01:12:05.363644 599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:05.365065 424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:12:05.365222 424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/wal
uuid: "a749f9fa01ae42248b34e414f8c225c0"
format_stamp: "Formatted at 2026-08-12 01:12:05 on dist-test-slave-21b9"
server_key: "f8b31792fe478fe94371e643c5dad8dc"
server_key_iv: "1f0441cd41f3ed56ec80a4f3d37f11e2"
server_key_version: "encryptionkey@0"
I20260812 01:12:05.365484 424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:05.389011 424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:05.393893 424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:05.394146 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:05.425563 424 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.60:34219
I20260812 01:12:05.425643 688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.60:34219 every 8 connection(s)
I20260812 01:12:05.427122 424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
I20260812 01:12:05.429806 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 424
I20260812 01:12:05.432495 691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:05.443127 691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:12:05 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.151.60@KRBTEST.COM
Aug 12 01:12:05 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497125, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.60@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:05.458177 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.432389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.20.151.60:34219)
0812 01:12:05.433760 (+ 1371us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:05.433772 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:05.433814 (+ 42us) server_negotiation.cc:407] Connection header received
0812 01:12:05.436184 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:05.436209 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:05.436598 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:05.436983 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:05.438879 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.440440 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.441728 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.442552 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.442726 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.449005 (+ 6279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:05.449016 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:05.449027 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:05.449064 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:05.452703 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.453599 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.453607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.453615 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.453906 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.454477 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.454482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.454486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.455422 (+ 936us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:05.455936 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:05.456235 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 01:12:05.456626 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1168,"thread_start_us":803,"threads_started":1}
I20260812 01:12:05.464960 697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.446220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.20.151.62:33689)
0812 01:12:05.446957 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:05.446964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:05.447798 (+ 834us) server_negotiation.cc:407] Connection header received
0812 01:12:05.448039 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:05.448043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:05.448111 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:05.448192 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:05.449987 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.450735 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.452648 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.453178 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.453316 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.457767 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:05.457772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:05.457776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:05.457811 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:05.461685 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.462684 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.462690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.462694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.462837 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.463217 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.463223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.463227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.463568 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:05.463707 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:05.464269 (+ 562us) server_negotiation.cc:299] Negotiation successful
0812 01:12:05.464506 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":454,"threads_started":1}
I20260812 01:12:05.464987 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.446288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:41024)
0812 01:12:05.447653 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:05.447703 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:05.447883 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:05.448303 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:05.448312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:05.449264 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:05.449824 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.449836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.450890 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.450895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.452499 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.452513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.453389 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.453394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.453541 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.454318 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:05.454340 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:05.457563 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:05.461812 (+ 4249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.461824 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.461829 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.462543 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.462957 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.462961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.462963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.463100 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.463698 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:05.463706 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:05.463961 (+ 255us) client_negotiation.cc:769] Sending connection context
0812 01:12:05.464141 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 01:12:05.464497 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":751,"threads_started":1}
I20260812 01:12:05.466480 691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:12:05 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497125, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.60@KRBTEST.COM for oryx/127.20.151.61@KRBTEST.COM
I20260812 01:12:05.481313 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.467475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.20.151.61:40015)
0812 01:12:05.467647 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:05.467651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:05.467667 (+ 16us) server_negotiation.cc:407] Connection header received
0812 01:12:05.467718 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:05.467721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:05.467780 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:05.467860 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:05.468627 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.469414 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.470965 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.471514 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.471621 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.475343 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:05.475349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:05.475353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:05.475387 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:05.478655 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.479339 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.479345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.479348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.479560 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.479932 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.479936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.479938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.480213 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:05.480346 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:05.480724 (+ 378us) server_negotiation.cc:299] Negotiation successful
0812 01:12:05.480952 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 01:12:05.481341 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.467138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:40015 (local address 127.0.0.1:57910)
0812 01:12:05.467341 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:05.467358 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:05.467441 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:05.467897 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:05.467900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:05.468344 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:05.468490 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.468498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.469550 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.469554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.470820 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.470833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.471638 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.471643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.471774 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.472673 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:05.472691 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:05.475089 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:05.478793 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.478799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.478804 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.479211 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.479678 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.479683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.479687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.479827 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.480327 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:05.480332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:05.480474 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 01:12:05.480594 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 01:12:05.480913 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 01:12:05.482471 691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:12:05 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497125, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.60@KRBTEST.COM for oryx/127.20.151.60@KRBTEST.COM
I20260812 01:12:05.497134 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.483431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:34219 (local address 127.0.0.1:53760)
0812 01:12:05.483572 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:05.483592 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:05.483699 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:05.483962 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:05.483966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:05.484421 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:05.484573 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.484581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.485724 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.485729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.486751 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.486764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.487723 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.487731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.487909 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.488487 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:05.488505 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:05.490903 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:05.494309 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.494314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.494318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.494769 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.495114 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.495118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.495122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.495423 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.495932 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:05.495937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:05.496179 (+ 242us) client_negotiation.cc:769] Sending connection context
0812 01:12:05.496309 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 01:12:05.496578 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 01:12:05.497213 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.483348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.20.151.60:34219)
0812 01:12:05.483574 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:05.483579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:05.483650 (+ 71us) server_negotiation.cc:407] Connection header received
0812 01:12:05.483790 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:05.483795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:05.483859 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:05.483933 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:05.484698 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.485604 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.486881 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.487536 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.487645 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.491071 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:05.491076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:05.491079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:05.491108 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:05.494168 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.494880 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.494885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.494887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.495009 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.495533 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.495538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.495540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.495815 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:05.495959 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:05.496409 (+ 450us) server_negotiation.cc:299] Negotiation successful
0812 01:12:05.496741 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":63}
I20260812 01:12:05.502943 691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Bootstrap starting.
I20260812 01:12:05.506793 691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:05.508345 691 log.cc:824] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:05.512084 691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: No bootstrap required, opened a new log
I20260812 01:12:05.521392 691 raft_consensus.cc:348] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } }
I20260812 01:12:05.521819 691 raft_consensus.cc:374] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:05.521967 691 raft_consensus.cc:729] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a749f9fa01ae42248b34e414f8c225c0, State: Initialized, Role: FOLLOWER
I20260812 01:12:05.522644 691 consensus_queue.cc:260] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } }
I20260812 01:12:05.523922 702 sys_catalog.cc:455] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } } }
I20260812 01:12:05.524365 702 sys_catalog.cc:458] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:12:05.524921 691 sys_catalog.cc:564] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:12:05.530957 702 raft_consensus.cc:482] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:12:05.531404 702 raft_consensus.cc:504] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } }
I20260812 01:12:05.533264 702 leader_election.cc:288] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3bef72f2d2436dbd3f6506788f0b68 (127.20.151.62:33689), 9d65031b3e3745329fdb09dcf3882d61 (127.20.151.61:40015)
I20260812 01:12:05.534121 31934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a749f9fa01ae42248b34e414f8c225c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" is_pre_election: true
I20260812 01:12:05.534433 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a749f9fa01ae42248b34e414f8c225c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d65031b3e3745329fdb09dcf3882d61" is_pre_election: true
I20260812 01:12:05.535439 603 leader_election.cc:302] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a749f9fa01ae42248b34e414f8c225c0; no voters: 9d65031b3e3745329fdb09dcf3882d61, cd3bef72f2d2436dbd3f6506788f0b68
I20260812 01:12:05.535902 702 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260812 01:12:05.540217 719 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:12:05.540374 719 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:12:05.542200 424 master_runner.cc:428] Detected that this master a749f9fa01ae42248b34e414f8c225c0 is joining an existing cluster
I20260812 01:12:05.542449 424 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.151.60:34219
I20260812 01:12:05.565207 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 32022
I20260812 01:12:05.569432 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.548329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:34219 (local address 127.0.0.1:53768)
0812 01:12:05.549319 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:05.549339 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:05.549514 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:05.549916 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:05.549921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:05.551494 (+ 1573us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:05.552053 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.552069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.553907 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.553913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.556311 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.556328 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.557950 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.557957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.558224 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.559062 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:05.559092 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:05.560474 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:05.565917 (+ 5443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.565922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.565925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.566454 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.566944 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.566948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.566952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.567124 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.567855 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:05.567861 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:05.568035 (+ 174us) client_negotiation.cc:769] Sending connection context
0812 01:12:05.568285 (+ 250us) client_negotiation.cc:240] Negotiation successful
0812 01:12:05.568750 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":26,"thread_start_us":606,"threads_started":1}
I20260812 01:12:05.570947 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.548193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.20.151.60:34219)
0812 01:12:05.548414 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:05.548420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:05.549455 (+ 1035us) server_negotiation.cc:407] Connection header received
0812 01:12:05.549650 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:05.549655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:05.549747 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:05.549862 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:05.552234 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.553675 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.556501 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.557485 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.557616 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.560681 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:05.560694 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:05.560699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:05.560739 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:05.565792 (+ 5053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.566633 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.566638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.566642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.566816 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.567262 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.567290 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.567293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.567715 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:05.569978 (+ 2263us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:05.570122 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 01:12:05.570361 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:12:05.573791 697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.548193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.20.151.62:33689)
0812 01:12:05.548659 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:05.548663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:05.551592 (+ 2929us) server_negotiation.cc:407] Connection header received
0812 01:12:05.551877 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:05.551883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:05.552012 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:05.552084 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:05.554272 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.555793 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.560677 (+ 4884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.561344 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.561465 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.564841 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:05.564850 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:05.564855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:05.564899 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:05.568961 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.570333 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.570338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.570342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.570581 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.571677 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.571682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.571686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.572084 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:05.572237 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:05.573063 (+ 826us) server_negotiation.cc:299] Negotiation successful
0812 01:12:05.573357 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358}
I20260812 01:12:05.575546 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.549706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:41028)
0812 01:12:05.551438 (+ 1732us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:05.551455 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:05.551698 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:05.552295 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:05.552300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:05.553484 (+ 1184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:05.554083 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.554101 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.556077 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.556083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.558785 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.558832 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.561613 (+ 2781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.561622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.561967 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.562582 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:05.562599 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:05.564576 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:05.569226 (+ 4650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.569231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.569234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.570128 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.570728 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.570732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.570736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.570997 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.572322 (+ 1325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:05.572327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:05.572603 (+ 276us) client_negotiation.cc:769] Sending connection context
0812 01:12:05.574274 (+ 1671us) client_negotiation.cc:240] Negotiation successful
0812 01:12:05.574764 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1587,"mutex_wait_us":1,"thread_start_us":1297,"threads_started":1}
I20260812 01:12:05.592373 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.552067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.20.151.61:40015)
0812 01:12:05.552259 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:05.552264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:05.558937 (+ 6673us) server_negotiation.cc:407] Connection header received
0812 01:12:05.559243 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:05.559250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:05.559363 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:05.559440 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:05.573656 (+ 14216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.575031 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.577826 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.578504 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.578649 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.582115 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:05.582123 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:05.582128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:05.582174 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:05.586817 (+ 4643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.589101 (+ 2284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.589107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.589112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.589350 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.589940 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.589945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.589949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.590330 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:05.590472 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:05.591350 (+ 878us) server_negotiation.cc:299] Negotiation successful
0812 01:12:05.591665 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 01:12:05.593498 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.552261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:40015 (local address 127.0.0.1:57920)
0812 01:12:05.558802 (+ 6541us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:05.558823 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:05.559080 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:05.559718 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:05.559723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:05.561737 (+ 2014us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:05.573078 (+ 11341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.573092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.575172 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.575178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.577092 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.577107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.579014 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.579021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.579334 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.580444 (+ 1110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:05.580468 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:05.581839 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:05.588237 (+ 6398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.588242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.588246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.588926 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.589521 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.589525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.589529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.589794 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.590566 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:05.590570 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:05.590894 (+ 324us) client_negotiation.cc:769] Sending connection context
0812 01:12:05.592673 (+ 1779us) client_negotiation.cc:240] Negotiation successful
0812 01:12:05.593032 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6311,"thread_start_us":6036,"threads_started":1}
Aug 12 01:12:05 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497122, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.62@KRBTEST.COM for oryx/127.20.151.60@KRBTEST.COM
Aug 12 01:12:05 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.151.1@KRBTEST.COM
I20260812 01:12:05.605230 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.582601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:34219 (local address 127.0.0.1:53780)
0812 01:12:05.583065 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:05.583085 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:05.583189 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:05.585438 (+ 2249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:05.585444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:05.585971 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:05.586290 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.586305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.588234 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.588240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.589615 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.589631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.591075 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.591081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.591246 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.591829 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:05.591847 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:05.596912 (+ 5065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:05.601827 (+ 4915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.601833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.601836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.602328 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.602702 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.602708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.602711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.602939 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.603806 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:05.603811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:05.604048 (+ 237us) client_negotiation.cc:769] Sending connection context
0812 01:12:05.604213 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 01:12:05.604659 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309}
I20260812 01:12:05.605719 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.567314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.20.151.1:33281)
0812 01:12:05.582114 (+ 14800us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:05.582125 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:05.582160 (+ 35us) server_negotiation.cc:407] Connection header received
0812 01:12:05.582235 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:05.582244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:05.582552 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:05.583080 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:05.584829 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.586572 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.588482 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.589597 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.597142 (+ 7545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.598031 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:05.598044 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:05.598060 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:05.598098 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:05.601850 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.602778 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.602789 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.602793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.603018 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.603541 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.603546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.603549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.604001 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:05.604331 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:05.604772 (+ 441us) server_negotiation.cc:299] Negotiation successful
0812 01:12:05.605073 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14557,"thread_start_us":15423,"threads_started":1}
I20260812 01:12:05.606427 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.583435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.20.151.60:34219)
0812 01:12:05.584989 (+ 1554us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:05.584995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:05.585049 (+ 54us) server_negotiation.cc:407] Connection header received
0812 01:12:05.585153 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:05.585159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:05.585256 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:05.586447 (+ 1191us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:05.586685 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.588054 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.589804 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.590869 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.591587 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.597172 (+ 5585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:05.597180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:05.597186 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:05.597225 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:05.601629 (+ 4404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.602450 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.602454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.602457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.602576 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.603081 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.603087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.603091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.603657 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:05.603803 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:05.605701 (+ 1898us) server_negotiation.cc:299] Negotiation successful
0812 01:12:05.605976 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410}
I20260812 01:12:05.607461 32218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:05.608203 31917 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.151.60:34219: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0"
member_type: NON_VOTER
last_known_addr {
host: "127.20.151.60"
port: 34219
}
attrs {
promote: true
}
}
dest_uuid: "cd3bef72f2d2436dbd3f6506788f0b68"
cas_config_opid_index: 5
I20260812 01:12:05.609499 31917 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } }
I20260812 01:12:05.612006 410 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a749f9fa01ae42248b34e414f8c225c0 (127.20.151.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
I20260812 01:12:05.616499 501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 LEARNER]: Refusing update from remote peer cd3bef72f2d2436dbd3f6506788f0b68: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 01:12:05.618569 646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:05.618918 410 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.151.60:34219
I20260812 01:12:05.623565 646 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 FOLLOWER]: Refusing update from remote peer cd3bef72f2d2436dbd3f6506788f0b68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 01:12:05.628034 742 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260812 01:12:05.638053 744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 01:12:05.640108 742 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
I20260812 01:12:05.640796 742 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:05.643736 424 master.cc:537] Master@127.20.151.60:34219 shutting down...
I20260812 01:12:05.644655 501 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a749f9fa01ae42248b34e414f8c225c0 (127.20.151.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
W20260812 01:12:05.647531 608 tablet.cc:2369] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:12:05.650274 550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
I20260812 01:12:05.650636 550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: This master's current role is: LEARNER
I20260812 01:12:05.651789 742 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68: attempting to promote NON_VOTER 9d65031b3e3745329fdb09dcf3882d61 to VOTER
I20260812 01:12:05.652870 742 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } }
I20260812 01:12:05.655963 497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 LEARNER]: Refusing update from remote peer cd3bef72f2d2436dbd3f6506788f0b68: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 01:12:05.656682 742 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 01:12:05.659401 744 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9d65031b3e3745329fdb09dcf3882d61 (127.20.151.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
W20260812 01:12:05.660532 31886 consensus_peers.cc:487] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 -> Peer a749f9fa01ae42248b34e414f8c225c0 (127.20.151.60:34219): Couldn't send request to peer a749f9fa01ae42248b34e414f8c225c0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 01:12:05.664196 497 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9d65031b3e3745329fdb09dcf3882d61 (127.20.151.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
I20260812 01:12:05.664558 742 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
I20260812 01:12:05.664903 742 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:05.665200 742 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
I20260812 01:12:05.665455 742 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:05.665997 646 raft_consensus.cc:858] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } }
W20260812 01:12:05.666366 646 raft_consensus.cc:1565] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } }
I20260812 01:12:05.668167 550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
I20260812 01:12:05.668421 550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:12:05.676829 646 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9d65031b3e3745329fdb09dcf3882d61 (127.20.151.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } }
I20260812 01:12:05.682444 755 sys_catalog.cc:455] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } }
I20260812 01:12:05.682955 755 sys_catalog.cc:458] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 01:12:05.683255 755 sys_catalog.cc:455] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } }
I20260812 01:12:05.683490 755 sys_catalog.cc:458] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 01:12:05.683904 702 sys_catalog.cc:455] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd3bef72f2d2436dbd3f6506788f0b68. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: NON_VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: true } } }
I20260812 01:12:05.684199 702 sys_catalog.cc:458] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 01:12:05.686894 424 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260812 01:12:05.687388 424 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260812 01:12:05.687592 424 tablet_replica.cc:330] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: stopping tablet replica
I20260812 01:12:05.695101 424 master.cc:552] Master@127.20.151.60:34219 shutdown complete.
I20260812 01:12:05.695245 424 master_runner.cc:305] Clearing existing system tablet
I20260812 01:12:05.695724 424 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:05.695886 424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:05.698863 424 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 01:12:05.700592 764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:05.700945 424 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:05.701371 424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260812 01:12:05.701521 424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/wal
uuid: "a749f9fa01ae42248b34e414f8c225c0"
format_stamp: "Formatted at 2026-08-12 01:12:05 on dist-test-slave-21b9"
server_key: "f8b31792fe478fe94371e643c5dad8dc"
server_key_iv: "1f0441cd41f3ed56ec80a4f3d37f11e2"
server_key_version: "encryptionkey@0"
I20260812 01:12:05.706429 424 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:05.715921 424 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 01:12:05.716333 424 log.cc:1196] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 01:12:05.717273 424 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Deleting consensus metadata
I20260812 01:12:05.717633 424 master_runner.cc:315] Copying system tablet from 127.20.151.62:33689
I20260812 01:12:05.720605 424 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: tablet copy: Beginning tablet copy session from remote peer at address 127.20.151.62:33689
I20260812 01:12:05.740914 697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.721610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.20.151.62:33689)
0812 01:12:05.721853 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:05.721860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:05.724233 (+ 2373us) server_negotiation.cc:407] Connection header received
0812 01:12:05.724592 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:05.724600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:05.724718 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:05.724829 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:05.726181 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.727393 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.729484 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.730218 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.730353 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.733871 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:05.733879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:05.733884 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:05.733929 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:05.737589 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.738426 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.738440 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.738444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.738659 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:05.739100 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:05.739105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:05.739109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:05.739536 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:05.739695 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:05.740206 (+ 511us) server_negotiation.cc:299] Negotiation successful
0812 01:12:05.740464 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260812 01:12:05.741396 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:05.721595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:41042)
0812 01:12:05.724078 (+ 2483us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:05.724098 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:05.724372 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:05.724865 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:05.724870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:05.725639 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:05.725967 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.725981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.727554 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.727559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.729314 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:05.729330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:05.730382 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:05.730388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:05.730649 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:05.732041 (+ 1392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:05.732066 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:05.733572 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:05.737755 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.737762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.737767 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.738291 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.738788 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:05.738793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:05.738797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:05.738981 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:05.739669 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:05.739674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:05.739872 (+ 198us) client_negotiation.cc:769] Sending connection context
0812 01:12:05.740030 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 01:12:05.740429 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2335,"thread_start_us":2157,"threads_started":1}
I20260812 01:12:05.742152 31951 tablet_copy_service.cc:140] P cd3bef72f2d2436dbd3f6506788f0b68: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a749f9fa01ae42248b34e414f8c225c0 ({username='oryx', principal='oryx/127.20.151.60@KRBTEST.COM'} at 127.0.0.1:41042)
I20260812 01:12:05.742376 31951 tablet_copy_service.cc:161] P cd3bef72f2d2436dbd3f6506788f0b68: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a749f9fa01ae42248b34e414f8c225c0 at {username='oryx', principal='oryx/127.20.151.60@KRBTEST.COM'} at 127.0.0.1:41042: session id = a749f9fa01ae42248b34e414f8c225c0-00000000000000000000000000000000
I20260812 01:12:05.747015 31951 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 01:12:05.749255 424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 01:12:05.757946 424 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: tablet copy: Starting download of 0 data blocks...
I20260812 01:12:05.758419 424 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: tablet copy: Starting download of 1 WAL segments...
I20260812 01:12:05.761601 424 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 01:12:05.767145 31951 tablet_copy_service.cc:342] P cd3bef72f2d2436dbd3f6506788f0b68: Request end of tablet copy session a749f9fa01ae42248b34e414f8c225c0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.151.60@KRBTEST.COM'} at 127.0.0.1:41042
I20260812 01:12:05.767347 31951 tablet_copy_service.cc:433] P cd3bef72f2d2436dbd3f6506788f0b68: ending tablet copy session a749f9fa01ae42248b34e414f8c225c0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a749f9fa01ae42248b34e414f8c225c0
I20260812 01:12:05.780305 424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:05.784109 780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:05.785079 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 32022
W20260812 01:12:05.794866 785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:05.807565 424 server_base.cc:1034] running on GCE node
W20260812 01:12:05.809005 787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:05.809661 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:05.810654 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:05.811949 424 hybrid_clock.cc:648] HybridClock initialized: now 1786497125811897 us; error 41 us; skew 500 ppm
I20260812 01:12:05.812197 424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:05 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497125, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:05.814486 424 init.cc:377] Logged in from keytab as oryx/127.20.151.60@KRBTEST.COM (short username oryx)
I20260812 01:12:05.818822 424 webserver.cc:460] Webserver started at http://127.20.151.60:36735/ using document root <none> and password file <none>
I20260812 01:12:05.818953 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:33281
--local_ip_for_outbound_sockets=127.20.151.1
--tserver_master_addrs=127.20.151.62:33689
--webserver_port=38069
--webserver_interface=127.20.151.1
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab
--principal=oryx/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.151.62:33689,127.20.151.61:40015
--tserver_master_addrs=127.20.151.62:33689,127.20.151.61:40015,127.20.151.60:34219 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:12:05.819188 424 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:05.819332 424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:05.822619 424 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260812 01:12:05.827718 800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:05.828197 424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260812 01:12:05.828351 424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/wal
uuid: "a749f9fa01ae42248b34e414f8c225c0"
format_stamp: "Formatted at 2026-08-12 01:12:05 on dist-test-slave-21b9"
server_key: "f8b31792fe478fe94371e643c5dad8dc"
server_key_iv: "1f0441cd41f3ed56ec80a4f3d37f11e2"
server_key_version: "encryptionkey@0"
I20260812 01:12:05.828562 424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:05.841272 424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:05.845726 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:05.884069 424 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.60:34219
I20260812 01:12:05.884140 886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.60:34219 every 8 connection(s)
I20260812 01:12:05.885128 424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
I20260812 01:12:05.890513 891 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:12:05.894398 891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Bootstrap starting.
I20260812 01:12:05.910808 891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:12:05.911578 891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Bootstrap complete.
I20260812 01:12:05.913439 891 raft_consensus.cc:348] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } }
I20260812 01:12:05.913800 891 raft_consensus.cc:729] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a749f9fa01ae42248b34e414f8c225c0, State: Initialized, Role: LEARNER
I20260812 01:12:05.914074 891 consensus_queue.cc:260] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } }
I20260812 01:12:05.915048 900 sys_catalog.cc:455] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
I20260812 01:12:05.915423 891 sys_catalog.cc:564] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:12:05.915567 900 sys_catalog.cc:458] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: This master's current role is: LEARNER
I20260812 01:12:05.932380 914 catalog_manager.cc:1259] Loaded cluster ID: 1c08e6e0a00e4060aacdd304bfe6627b
I20260812 01:12:05.932564 914 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: loading cluster ID for follower catalog manager: success
I20260812 01:12:05.936187 914 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: acquiring CA information for follower catalog manager: success
I20260812 01:12:05.938768 914 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:12:06.106416 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:06.099815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:34219 (local address 127.0.0.1:53782)
0812 01:12:06.100003 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:06.100025 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:06.100166 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:06.100976 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:06.100983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:06.101017 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:06.101216 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:06.101227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.102536 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.102542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:06.104375 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:06.104423 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.105585 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.105590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:06.105829 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:06.105862 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 01:12:06.105947 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 01:12:06.106087 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 01:12:06.106480 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:06.100071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.20.151.60:34219)
0812 01:12:06.100649 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:06.100653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:06.100674 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:12:06.100740 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:06.100743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:06.100812 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:06.100910 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:06.101365 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.102371 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.104592 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.105423 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.105540 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:06.105795 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:06.105991 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 01:12:06.106132 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":314,"threads_started":1}
I20260812 01:12:06.107358 850 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 01:12:06.107815 900 sys_catalog.cc:455] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd3bef72f2d2436dbd3f6506788f0b68. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
I20260812 01:12:06.108089 900 sys_catalog.cc:458] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: This master's current role is: LEARNER
W20260812 01:12:06.241593 801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:06.242012 801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:06.242272 801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:06.251820 801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:06.252030 801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:06.252292 801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:12:06.264159 801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab
--principal=oryx/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:33281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=38069
--tserver_master_addrs=127.20.151.62:33689,127.20.151.61:40015,127.20.151.60:34219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:06.265741 801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:06.267251 801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:06.282500 928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:06.283344 937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:06.283663 933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:06.284247 801 server_base.cc:1034] running on GCE node
I20260812 01:12:06.284966 801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:06.286157 801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:06.287474 801 hybrid_clock.cc:648] HybridClock initialized: now 1786497126287336 us; error 44 us; skew 500 ppm
I20260812 01:12:06.287941 801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:06 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497126, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:06.292513 801 init.cc:377] Logged in from keytab as oryx/127.20.151.1@KRBTEST.COM (short username oryx)
I20260812 01:12:06.294437 801 webserver.cc:460] Webserver started at http://127.20.151.1:38069/ using document root <none> and password file <none>
I20260812 01:12:06.295389 801 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:06.295629 801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:06.302311 801 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.005s
I20260812 01:12:06.305675 953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:06.307232 801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:12:06.307525 801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "dd7fce201fee4ca3b83deab3cb7ac345"
format_stamp: "Formatted at 2026-08-12 01:12:02 on dist-test-slave-21b9"
server_key: "806af2a5293b48ddd0ba0a452f021cfd"
server_key_iv: "e07a02bc8a9f2133dfb273670b13291c"
server_key_version: "encryptionkey@0"
I20260812 01:12:06.308686 801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:06.331956 801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:06.337860 801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:06.338146 801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:06.339663 801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:06.341459 801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:06.341593 801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:06.341727 801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:06.341863 801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:06.399967 801 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:33281
I20260812 01:12:06.400143 1160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:33281 every 8 connection(s)
I20260812 01:12:06.401698 801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:12:06.410856 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 801
I20260812 01:12:06.411198 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 32232
Aug 12 01:12:06 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497126, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
Aug 12 01:12:06 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497126, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.60@KRBTEST.COM
Aug 12 01:12:06 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497126, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.61@KRBTEST.COM
I20260812 01:12:06.434391 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:06.405466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:55283 (local address 127.20.151.60:34219)
0812 01:12:06.405677 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:06.405683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:06.408155 (+ 2472us) server_negotiation.cc:407] Connection header received
0812 01:12:06.411698 (+ 3543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:06.411705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:06.411799 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:06.411910 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:06.414934 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.416301 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.419154 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.419850 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.419932 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:06.426368 (+ 6436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:06.426376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:06.426381 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:06.426429 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:06.430213 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:06.431387 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:06.431395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:06.431398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:06.431560 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:06.432199 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:06.432205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:06.432208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:06.432567 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:06.432688 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:06.433724 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0812 01:12:06.433971 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 01:12:06.434427 1166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:06.405395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:33859 (local address 127.20.151.61:40015)
0812 01:12:06.406355 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:06.406362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:06.408133 (+ 1771us) server_negotiation.cc:407] Connection header received
0812 01:12:06.411802 (+ 3669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:06.411808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:06.411901 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:06.412010 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:06.416664 (+ 4654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.417939 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.420244 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.421058 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.421173 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:06.427060 (+ 5887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:06.427067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:06.427072 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:06.427115 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:06.430963 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:06.431981 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:06.431987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:06.431990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:06.432198 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:06.432645 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:06.432650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:06.432654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:06.432995 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:06.433127 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:06.433758 (+ 631us) server_negotiation.cc:299] Negotiation successful
0812 01:12:06.433967 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":657,"threads_started":1}
I20260812 01:12:06.434423 1179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:06.408684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:44183 (local address 127.20.151.62:33689)
0812 01:12:06.409980 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:06.409984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:06.410001 (+ 17us) server_negotiation.cc:407] Connection header received
0812 01:12:06.412173 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:06.412178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:06.412262 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:06.412368 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:06.416022 (+ 3654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.417100 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.419462 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.420156 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.420287 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:06.425917 (+ 5630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:06.425925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:06.425930 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:06.425977 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:06.430213 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:06.431501 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:06.431507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:06.431511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:06.431648 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:06.432179 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:06.432185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:06.432189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:06.432511 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:06.432673 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:06.433715 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0812 01:12:06.434051 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1180,"thread_start_us":1017,"threads_started":1}
I20260812 01:12:06.436501 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:06.408657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.1:44183)
0812 01:12:06.409745 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:06.409766 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:06.411601 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:06.412896 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:06.412901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:06.415623 (+ 2722us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:06.415880 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:06.415889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.417246 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.417250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:06.419332 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:06.419346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.420456 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.420462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:06.420773 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:06.421723 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:06.421767 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:06.425616 (+ 3849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:06.430650 (+ 5034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:06.430656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:06.430660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:06.431370 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:06.431768 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:06.431773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:06.431776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:06.432058 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:06.432659 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:06.432667 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:06.433174 (+ 507us) client_negotiation.cc:769] Sending connection context
0812 01:12:06.433605 (+ 431us) client_negotiation.cc:240] Negotiation successful
0812 01:12:06.434172 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":119,"spinlock_wait_cycles":17664,"thread_start_us":753,"threads_started":1}
I20260812 01:12:06.436578 1173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:06.406193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:40015 (local address 127.20.151.1:33859)
0812 01:12:06.407924 (+ 1731us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:06.407977 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:06.411314 (+ 3337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:06.412244 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:06.412248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:06.416200 (+ 3952us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:06.416495 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:06.416508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.418154 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.418159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:06.420059 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:06.420076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.422060 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.422067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:06.422264 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:06.423148 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:06.423176 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:06.426796 (+ 3620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:06.431141 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:06.431146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:06.431149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:06.431832 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:06.432342 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:06.432347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:06.432350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:06.432500 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:06.433206 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:06.433210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:06.433435 (+ 225us) client_negotiation.cc:769] Sending connection context
0812 01:12:06.434680 (+ 1245us) client_negotiation.cc:240] Negotiation successful
0812 01:12:06.435198 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1305,"mutex_wait_us":178,"spinlock_wait_cycles":10880,"thread_start_us":1057,"threads_started":1}
I20260812 01:12:06.436652 1167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:06.406193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:34219 (local address 127.20.151.1:55283)
0812 01:12:06.408027 (+ 1834us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:06.408047 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:06.411314 (+ 3267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:06.412235 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:06.412248 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:06.413825 (+ 1577us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:06.414733 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:06.414754 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.416514 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.416518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:06.418967 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:06.418982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.420410 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.420419 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:06.420778 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:06.421815 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:06.421838 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:06.426123 (+ 4285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:06.430390 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:06.430402 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:06.430423 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:06.431192 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:06.431727 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:06.431732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:06.431734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:06.432022 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:06.432780 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:06.432784 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:06.433169 (+ 385us) client_negotiation.cc:769] Sending connection context
0812 01:12:06.434680 (+ 1511us) client_negotiation.cc:240] Negotiation successful
0812 01:12:06.434977 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1608,"mutex_wait_us":184,"spinlock_wait_cycles":59008,"thread_start_us":989,"threads_started":1}
I20260812 01:12:06.438918 1161 heartbeater.cc:347] Connected to a master server at 127.20.151.60:34219
I20260812 01:12:06.439047 1164 heartbeater.cc:347] Connected to a master server at 127.20.151.61:40015
I20260812 01:12:06.439353 1161 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:06.439356 1164 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:06.438910 1162 heartbeater.cc:347] Connected to a master server at 127.20.151.62:33689
I20260812 01:12:06.439759 1162 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:06.440811 1164 heartbeater.cc:511] Master 127.20.151.61:40015 requested a full tablet report, sending...
I20260812 01:12:06.441021 1161 heartbeater.cc:511] Master 127.20.151.60:34219 requested a full tablet report, sending...
I20260812 01:12:06.441030 1162 heartbeater.cc:511] Master 127.20.151.62:33689 requested a full tablet report, sending...
Aug 12 01:12:06 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.151.2@KRBTEST.COM
I20260812 01:12:06.443097 31917 ts_manager.cc:194] Re-registered known tserver with Master: dd7fce201fee4ca3b83deab3cb7ac345 (127.20.151.1:33281)
I20260812 01:12:06.443723 481 ts_manager.cc:194] Registered new tserver with Master: dd7fce201fee4ca3b83deab3cb7ac345 (127.20.151.1:33281)
I20260812 01:12:06.444200 31917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:44183
I20260812 01:12:06.446158 828 ts_manager.cc:194] Registered new tserver with Master: dd7fce201fee4ca3b83deab3cb7ac345 (127.20.151.1:33281)
I20260812 01:12:06.451975 1184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:06.412600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.20.151.2:34343)
0812 01:12:06.413258 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:06.413267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:06.413511 (+ 244us) server_negotiation.cc:407] Connection header received
0812 01:12:06.413609 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:06.413618 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:06.413853 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:06.414040 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:06.417793 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.420644 (+ 2851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.422855 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:06.424024 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:06.426083 (+ 2059us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:06.443265 (+ 17182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:06.443325 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:06.443344 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:06.443399 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:06.448781 (+ 5382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:06.449842 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:06.449850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:06.449853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:06.450043 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:06.450363 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:06.450366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:06.450368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:06.450707 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:06.450928 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:06.451161 (+ 233us) server_negotiation.cc:299] Negotiation successful
0812 01:12:06.451447 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":313,"threads_started":1}
I20260812 01:12:06.453296 32456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:06.539418 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 32232
W20260812 01:12:06.561414 31891 connection.cc:535] server connection from 127.20.151.2:47763 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 01:12:06.562417 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:34343
--local_ip_for_outbound_sockets=127.20.151.2
--tserver_master_addrs=127.20.151.62:33689
--webserver_port=34561
--webserver_interface=127.20.151.2
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab
--principal=oryx/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.151.62:33689,127.20.151.61:40015
--tserver_master_addrs=127.20.151.62:33689,127.20.151.61:40015,127.20.151.60:34219 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:12:06.719223 744 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: NON_VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: true } } }
I20260812 01:12:06.719612 744 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:06.720813 744 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68: attempting to promote NON_VOTER a749f9fa01ae42248b34e414f8c225c0 to VOTER
I20260812 01:12:06.721457 744 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } }
I20260812 01:12:06.723905 850 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 LEARNER]: Refusing update from remote peer cd3bef72f2d2436dbd3f6506788f0b68: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 01:12:06.723986 497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 FOLLOWER]: Refusing update from remote peer cd3bef72f2d2436dbd3f6506788f0b68: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 01:12:06.724565 1189 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 01:12:06.725046 744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 01:12:06.728039 1189 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a749f9fa01ae42248b34e414f8c225c0 (127.20.151.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:06.729418 497 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a749f9fa01ae42248b34e414f8c225c0 (127.20.151.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:06.737223 744 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:06.737591 744 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:06.737972 744 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:06.738245 744 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:06.737496 1198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:06.738579 1198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:12:06.740064 744 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:06.740290 744 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:06.741217 850 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a749f9fa01ae42248b34e414f8c225c0 (127.20.151.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:06.744899 1199 sys_catalog.cc:455] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:06.745203 1199 sys_catalog.cc:458] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 01:12:06.917851 1187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:06.918182 1187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:06.918339 1187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:06.925869 1187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:06.926033 1187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:06.926196 1187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:12:06.936851 1187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab
--principal=oryx/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:34343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=34561
--tserver_master_addrs=127.20.151.62:33689,127.20.151.61:40015,127.20.151.60:34219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:06.938269 1187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:06.939744 1187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:06.953699 1217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:06.953652 1210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:06.953745 1209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:06.956709 1187 server_base.cc:1034] running on GCE node
I20260812 01:12:06.957401 1187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:06.958453 1187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:06.959734 1187 hybrid_clock.cc:648] HybridClock initialized: now 1786497126959665 us; error 55 us; skew 500 ppm
I20260812 01:12:06.960100 1187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:06 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497126, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:06.963312 1187 init.cc:377] Logged in from keytab as oryx/127.20.151.2@KRBTEST.COM (short username oryx)
I20260812 01:12:06.964879 1187 webserver.cc:460] Webserver started at http://127.20.151.2:34561/ using document root <none> and password file <none>
I20260812 01:12:06.965615 1187 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:06.965752 1187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:06.971418 1187 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 01:12:06.974830 1224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:06.976605 1187 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 01:12:06.976789 1187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "1955a48e1490429484d27bf69dd64ab5"
format_stamp: "Formatted at 2026-08-12 01:12:03 on dist-test-slave-21b9"
server_key: "95bac1fe70581a2ff690e9df74a8fe2e"
server_key_iv: "0378ed05156cbd3f0b2bf0bd28151ac4"
server_key_version: "encryptionkey@0"
I20260812 01:12:06.978219 1187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:07.000870 1187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:07.005250 1187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:07.005503 1187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:07.006523 1187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:07.008020 1187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:07.008121 1187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:07.008211 1187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:07.008281 1187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:07.043493 1187 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:34343
I20260812 01:12:07.043555 1433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:34343 every 8 connection(s)
I20260812 01:12:07.045130 1187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:12:07.051229 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 1187
I20260812 01:12:07.051551 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 32481
Aug 12 01:12:07 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.151.3@KRBTEST.COM
Aug 12 01:12:07 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497126, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.61@KRBTEST.COM
Aug 12 01:12:07 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497126, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
Aug 12 01:12:07 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497126, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.60@KRBTEST.COM
I20260812 01:12:07.089432 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.053161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.20.151.3:45441)
0812 01:12:07.060457 (+ 7296us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.060469 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.061392 (+ 923us) server_negotiation.cc:407] Connection header received
0812 01:12:07.061609 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.061622 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.061976 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.062153 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:07.065069 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.067021 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.069276 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.070288 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.070389 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.077788 (+ 7399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:07.077808 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:07.077825 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:07.077877 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:07.083380 (+ 5503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.086122 (+ 2742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.086141 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.086145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.086416 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.087031 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.087037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.087039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.087573 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:07.087905 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.088616 (+ 711us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.088952 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7088,"thread_start_us":6903,"threads_started":1}
I20260812 01:12:07.092360 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.048264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:49339 (local address 127.20.151.61:40015)
0812 01:12:07.050320 (+ 2056us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.050328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.064963 (+ 14635us) server_negotiation.cc:407] Connection header received
0812 01:12:07.069413 (+ 4450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.069419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.069503 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.069582 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:07.072669 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.074108 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.076559 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.077238 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.077360 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.083015 (+ 5655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:07.083021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:07.083025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:07.083063 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:07.088087 (+ 5024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.089089 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.089094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.089097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.089280 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.089913 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.089919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.089922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.090302 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:07.090441 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.091593 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.091817 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1900,"thread_start_us":1735,"threads_started":1}
I20260812 01:12:07.092384 32684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:07.093931 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.066740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:38723 (local address 127.20.151.62:33689)
0812 01:12:07.067248 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.067254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.068269 (+ 1015us) server_negotiation.cc:407] Connection header received
0812 01:12:07.069418 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.069425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.069529 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.069623 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:07.072692 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.073569 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.076411 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.077078 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.077208 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.085221 (+ 8013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:07.085229 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:07.085233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:07.085283 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:07.089310 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.090235 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.090241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.090244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.090483 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.091094 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.091099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.091102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.091505 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:07.091673 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.093197 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.093464 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
I20260812 01:12:07.094077 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.048367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:55197 (local address 127.20.151.60:34219)
0812 01:12:07.065583 (+ 17216us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.065590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.066720 (+ 1130us) server_negotiation.cc:407] Connection header received
0812 01:12:07.069512 (+ 2792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.069518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.069600 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.070578 (+ 978us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:07.072588 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.075304 (+ 2716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.078559 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.079196 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.079440 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.085530 (+ 6090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:07.085539 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:07.085543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:07.085588 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:07.089977 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.090888 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.090893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.090896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.091132 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.091608 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.091613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.091616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.091978 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:07.092125 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.093354 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.093633 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17072,"thread_start_us":16920,"threads_started":1}
I20260812 01:12:07.094348 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.048919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:40015 (local address 127.20.151.2:49339)
0812 01:12:07.064763 (+ 15844us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:07.064826 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:07.068976 (+ 4150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:07.070079 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:07.070085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:07.071054 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:07.072455 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.072476 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.074322 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.074328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.076336 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.076351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.077557 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.077565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.077927 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.078793 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:07.078843 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:07.082671 (+ 3828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:07.088276 (+ 5605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.088286 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.088304 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.088938 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.089441 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.089445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.089449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.089769 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.090540 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:07.090549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:07.091105 (+ 556us) client_negotiation.cc:769] Sending connection context
0812 01:12:07.092637 (+ 1532us) client_negotiation.cc:240] Negotiation successful
0812 01:12:07.092934 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15354,"mutex_wait_us":129,"thread_start_us":16893,"threads_started":1}
I20260812 01:12:07.094682 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.067297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.2:38723)
0812 01:12:07.068156 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:07.068171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:07.069068 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:07.070037 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:07.070050 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:07.071314 (+ 1264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:07.072559 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.072572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.073751 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.073756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.076278 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.076293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.077818 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.077824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.078133 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.078984 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:07.079007 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:07.084919 (+ 5912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:07.089487 (+ 4568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.089492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.089496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.090077 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.090663 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.090668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.090671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.090937 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.091742 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:07.091746 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:07.091900 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 01:12:07.092062 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 01:12:07.092615 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":320,"thread_start_us":461,"threads_started":1}
I20260812 01:12:07.095626 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.048919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:34219 (local address 127.20.151.2:55197)
0812 01:12:07.066533 (+ 17614us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:07.066551 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:07.069065 (+ 2514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:07.070037 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:07.070050 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:07.071314 (+ 1264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:07.072340 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.072376 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.075488 (+ 3112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.075493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.078381 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.078396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.079508 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.079517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.079690 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.080253 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:07.080269 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:07.085330 (+ 5061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:07.090145 (+ 4815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.090150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.090153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.090742 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.091307 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.091311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.091315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.091483 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.092205 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:07.092210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:07.092370 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 01:12:07.092524 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 01:12:07.092811 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17438,"mutex_wait_us":140,"spinlock_wait_cycles":95104,"thread_start_us":14973,"threads_started":1}
I20260812 01:12:07.096133 1436 heartbeater.cc:347] Connected to a master server at 127.20.151.61:40015
I20260812 01:12:07.096196 1435 heartbeater.cc:347] Connected to a master server at 127.20.151.62:33689
I20260812 01:12:07.096562 1436 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:07.096581 1435 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:07.097160 1434 heartbeater.cc:347] Connected to a master server at 127.20.151.60:34219
I20260812 01:12:07.097344 1434 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:07.097998 1436 heartbeater.cc:511] Master 127.20.151.61:40015 requested a full tablet report, sending...
I20260812 01:12:07.098111 1434 heartbeater.cc:511] Master 127.20.151.60:34219 requested a full tablet report, sending...
I20260812 01:12:07.098652 1435 heartbeater.cc:511] Master 127.20.151.62:33689 requested a full tablet report, sending...
I20260812 01:12:07.099862 31917 ts_manager.cc:194] Re-registered known tserver with Master: 1955a48e1490429484d27bf69dd64ab5 (127.20.151.2:34343)
I20260812 01:12:07.100220 828 ts_manager.cc:194] Registered new tserver with Master: 1955a48e1490429484d27bf69dd64ab5 (127.20.151.2:34343)
I20260812 01:12:07.100929 31917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:38723
I20260812 01:12:07.102226 481 ts_manager.cc:194] Registered new tserver with Master: 1955a48e1490429484d27bf69dd64ab5 (127.20.151.2:34343)
I20260812 01:12:07.191190 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 32481
I20260812 01:12:07.216684 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:45441
--local_ip_for_outbound_sockets=127.20.151.3
--tserver_master_addrs=127.20.151.62:33689
--webserver_port=37547
--webserver_interface=127.20.151.3
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab
--principal=oryx/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.151.62:33689,127.20.151.61:40015
--tserver_master_addrs=127.20.151.62:33689,127.20.151.61:40015,127.20.151.60:34219 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:12:07.447335 1162 heartbeater.cc:503] Master 127.20.151.62:33689 was elected leader, sending a full tablet report...
W20260812 01:12:07.562301 1453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:07.562598 1453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:07.562748 1453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:07.570348 1453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:07.570492 1453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:07.570658 1453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:12:07.580552 1453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab
--principal=oryx/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:45441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=37547
--tserver_master_addrs=127.20.151.62:33689,127.20.151.61:40015,127.20.151.60:34219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:07.581903 1453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:07.583312 1453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:07.596529 1462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:07.596458 1464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:07.596479 1467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:07.597561 1453 server_base.cc:1034] running on GCE node
I20260812 01:12:07.598304 1453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:07.599442 1453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:07.600688 1453 hybrid_clock.cc:648] HybridClock initialized: now 1786497127600624 us; error 52 us; skew 500 ppm
I20260812 01:12:07.600997 1453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:07 dist-test-slave-21b9 krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497127, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:07.604131 1453 init.cc:377] Logged in from keytab as oryx/127.20.151.3@KRBTEST.COM (short username oryx)
I20260812 01:12:07.605453 1453 webserver.cc:460] Webserver started at http://127.20.151.3:37547/ using document root <none> and password file <none>
I20260812 01:12:07.606081 1453 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:07.606194 1453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:07.611370 1453 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:12:07.614163 1481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:07.615490 1453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 01:12:07.615657 1453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "eec8159f12d9469aa0168790d61e4bda"
format_stamp: "Formatted at 2026-08-12 01:12:03 on dist-test-slave-21b9"
server_key: "bbe27d84fb3321ea634b016d5ce73ca8"
server_key_iv: "b18bc0ce78d8e3e7dac32d96cbf8bbd2"
server_key_version: "encryptionkey@0"
I20260812 01:12:07.616593 1453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:07.660374 1453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:07.665577 1453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:07.665880 1453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:07.666922 1453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:07.668458 1453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:07.668550 1453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:07.668663 1453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:07.668740 1453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:07.700534 1453 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:45441
I20260812 01:12:07.700618 1685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:45441 every 8 connection(s)
I20260812 01:12:07.701992 1453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:12:07.704484 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 1453
Aug 12 01:12:07 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497127, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.60@KRBTEST.COM
Aug 12 01:12:07 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497127, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:07.742305 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.705625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:34219 (local address 127.20.151.3:35205)
0812 01:12:07.706534 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:07.706595 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:07.709410 (+ 2815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:07.712579 (+ 3169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:07.712591 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:07.713476 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:07.715506 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.715521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.722003 (+ 6482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.722009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.723442 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.723459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.724557 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.724565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.724782 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.725621 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:07.725655 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:07.729311 (+ 3656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:07.735864 (+ 6553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.735875 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.735896 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.736544 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.737939 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.737944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.737948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.738333 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.739031 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:07.739042 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:07.739574 (+ 532us) client_negotiation.cc:769] Sending connection context
0812 01:12:07.739914 (+ 340us) client_negotiation.cc:240] Negotiation successful
0812 01:12:07.740501 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":1278,"thread_start_us":612,"threads_started":1}
I20260812 01:12:07.744206 1700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.706260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:35205 (local address 127.20.151.60:34219)
0812 01:12:07.711962 (+ 5702us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.711968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.711990 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:12:07.712059 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.712064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.712149 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.720181 (+ 8032us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:07.720677 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.721828 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.723705 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.724339 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.725246 (+ 907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.729711 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:07.729719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:07.729725 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:07.729770 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:07.735712 (+ 5942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.737626 (+ 1914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.737632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.737635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.737806 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.738483 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.738488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.738491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.738870 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:07.743320 (+ 4450us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.743468 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.743738 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5577,"thread_start_us":5474,"threads_started":1}
Aug 12 01:12:07 dist-test-slave-21b9 krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497127, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.61@KRBTEST.COM
I20260812 01:12:07.745922 1686 heartbeater.cc:347] Connected to a master server at 127.20.151.60:34219
I20260812 01:12:07.746285 1686 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:07.747313 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.704942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:49493 (local address 127.20.151.62:33689)
0812 01:12:07.706037 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.706044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.712461 (+ 6417us) server_negotiation.cc:407] Connection header received
0812 01:12:07.712718 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.712723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.712847 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.712934 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:07.715127 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.716323 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.719459 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.720158 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.720283 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.737550 (+ 17267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:07.737558 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:07.737563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:07.737618 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:07.741302 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.742348 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.742354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.742357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.742954 (+ 597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.743711 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.743716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.743719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.744513 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:07.744668 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.746507 (+ 1839us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.746774 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":860,"threads_started":1}
I20260812 01:12:07.747419 1686 heartbeater.cc:511] Master 127.20.151.60:34219 requested a full tablet report, sending...
I20260812 01:12:07.748706 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.705625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.3:49493)
0812 01:12:07.712358 (+ 6733us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:07.712377 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:07.712563 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:07.713265 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:07.713269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:07.714005 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:07.714891 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.714928 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.716974 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.716981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.719244 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.719261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.720411 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.720419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.726839 (+ 6420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.728943 (+ 2104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:07.728972 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:07.737309 (+ 8337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:07.741436 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.741442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.741447 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.742196 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.743368 (+ 1172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.743373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.743376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.743580 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.745799 (+ 2219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:07.745809 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:07.746144 (+ 335us) client_negotiation.cc:769] Sending connection context
0812 01:12:07.747733 (+ 1589us) client_negotiation.cc:240] Negotiation successful
0812 01:12:07.748100 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6573,"mutex_wait_us":493,"thread_start_us":5833,"threads_started":1}
I20260812 01:12:07.749521 828 ts_manager.cc:194] Registered new tserver with Master: eec8159f12d9469aa0168790d61e4bda (127.20.151.3:45441)
I20260812 01:12:07.750707 1688 heartbeater.cc:347] Connected to a master server at 127.20.151.62:33689
I20260812 01:12:07.750855 1688 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:07.751505 1688 heartbeater.cc:511] Master 127.20.151.62:33689 requested a full tablet report, sending...
I20260812 01:12:07.752817 31917 ts_manager.cc:194] Re-registered known tserver with Master: eec8159f12d9469aa0168790d61e4bda (127.20.151.3:45441)
I20260812 01:12:07.754084 31917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:49493
I20260812 01:12:07.762609 1706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.712533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:40015 (local address 127.20.151.3:35753)
0812 01:12:07.719827 (+ 7294us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:07.719846 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:07.719971 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:07.721158 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:07.721162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:07.727893 (+ 6731us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:07.729084 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.729101 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.733687 (+ 4586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.733694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.735543 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.735562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.736773 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.736779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.736996 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.742555 (+ 5559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:07.742581 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:07.746052 (+ 3471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:07.759084 (+ 13032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.759091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.759095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.759804 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.760427 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:07.760433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:07.760437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:07.760610 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:07.761320 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:07.761325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:07.761487 (+ 162us) client_negotiation.cc:769] Sending connection context
0812 01:12:07.761635 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 01:12:07.762025 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7175,"thread_start_us":6957,"threads_started":1}
I20260812 01:12:07.763569 1707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.714268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:35753 (local address 127.20.151.61:40015)
0812 01:12:07.720866 (+ 6598us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.720872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.720890 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:12:07.720942 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.720946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.721005 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.721105 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:07.729450 (+ 8345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.733323 (+ 3873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.735698 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.736300 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.736426 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.748711 (+ 12285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:07.748718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:07.748722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:07.748770 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:07.758838 (+ 10068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.759969 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.759974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.759978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.760281 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.760748 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.760753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.760756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.761099 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:07.761253 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.762301 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.762521 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6478,"thread_start_us":6359,"threads_started":1}
I20260812 01:12:07.764556 1690 heartbeater.cc:347] Connected to a master server at 127.20.151.61:40015
I20260812 01:12:07.764725 1690 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:07.765129 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.722746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.20.151.60:34219)
0812 01:12:07.736403 (+ 13657us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.736410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.737850 (+ 1440us) server_negotiation.cc:407] Connection header received
0812 01:12:07.737900 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.737904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.737974 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.738082 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:07.747055 (+ 8973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.748193 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.750822 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.751667 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.751800 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.756676 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:07.756685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:07.756689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:07.756733 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:07.760743 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.761551 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.761557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.761560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.761819 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.762363 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.762368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.762371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.762793 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:07.763471 (+ 678us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.764429 (+ 958us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.764683 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13522,"thread_start_us":13661,"threads_started":1}
I20260812 01:12:07.765390 1690 heartbeater.cc:511] Master 127.20.151.61:40015 requested a full tablet report, sending...
I20260812 01:12:07.767016 481 ts_manager.cc:194] Registered new tserver with Master: eec8159f12d9469aa0168790d61e4bda (127.20.151.3:45441)
I20260812 01:12:07.768936 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.721987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.20.151.62:33689)
0812 01:12:07.733033 (+ 11046us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.733041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.733063 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:12:07.733133 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.733139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.733225 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.733340 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:07.747106 (+ 13766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.751764 (+ 4658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.755834 (+ 4070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.756539 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.756672 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.760645 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:07.760652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:07.760658 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:07.760698 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:07.765058 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.765970 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.765976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.765980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.766184 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.766939 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.766944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.766947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.767597 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:07.767793 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.768332 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.768571 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10875,"thread_start_us":10629,"threads_started":1}
I20260812 01:12:07.778045 1722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.738568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.20.151.61:40015)
0812 01:12:07.743297 (+ 4729us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.743303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.751764 (+ 8461us) server_negotiation.cc:407] Connection header received
0812 01:12:07.751835 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.751840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.752036 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.752147 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:07.756532 (+ 4385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.757884 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.760365 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.761140 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.761262 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.766701 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:07.766708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:07.766712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:07.766748 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:07.774657 (+ 7909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.775710 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.775715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.775717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.775938 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:07.776353 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:07.776357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:07.776359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:07.776643 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:07.776892 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.777275 (+ 383us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.777494 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4573,"thread_start_us":6486,"threads_started":1}
I20260812 01:12:07.775506 31917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:12:07.785249 31917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:12:07.831226 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.806874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:33281 (local address 127.0.0.1:40516)
0812 01:12:07.808395 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:07.808419 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:07.808617 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:07.821662 (+ 13045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:07.821668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:07.821708 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:07.822307 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.822326 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.824666 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.824672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.827412 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.827430 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.830100 (+ 2670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.830107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.830314 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.830348 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 01:12:07.830447 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 01:12:07.830602 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1363,"mutex_wait_us":22,"thread_start_us":1186,"threads_started":1}
I20260812 01:12:07.832064 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.806925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.20.151.1:33281)
0812 01:12:07.819957 (+ 13032us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.819967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.820010 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:12:07.820162 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.820172 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.820974 (+ 802us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.821269 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:07.822562 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.824487 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.827609 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.829551 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.829698 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.829930 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.831670 (+ 1740us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.831847 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12714,"thread_start_us":13345,"threads_started":1}
I20260812 01:12:07.833878 1754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.807012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:45441 (local address 127.0.0.1:41558)
0812 01:12:07.822606 (+ 15594us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:07.822632 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:07.823318 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:07.824129 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:07.824145 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:07.824172 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:07.825083 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.825098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.827504 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.827510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.830129 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.830153 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.832885 (+ 2732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.832892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.833194 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.833230 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 01:12:07.833329 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 01:12:07.833476 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15438,"mutex_wait_us":1,"thread_start_us":15994,"threads_started":1}
I20260812 01:12:07.834225 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.810165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:34343 (local address 127.0.0.1:43984)
0812 01:12:07.823346 (+ 13181us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:07.823365 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:07.823527 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:07.825367 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:07.825372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:07.825402 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:07.825917 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.825932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.828797 (+ 2865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.828801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.831111 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:07.831127 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.833423 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.833430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:07.833615 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.833658 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 01:12:07.833753 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 01:12:07.833894 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13003,"mutex_wait_us":3,"thread_start_us":11452,"threads_started":1}
I20260812 01:12:07.834522 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.807699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.20.151.3:45441)
0812 01:12:07.821422 (+ 13723us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.821432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.823488 (+ 2056us) server_negotiation.cc:407] Connection header received
0812 01:12:07.823609 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.823617 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.823911 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.824119 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:07.825400 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.827234 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.830722 (+ 3488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.832671 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.832808 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.833013 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.833698 (+ 685us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.833868 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13500,"thread_start_us":13226,"threads_started":1}
I20260812 01:12:07.834947 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:07.810855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.20.151.2:34343)
0812 01:12:07.819093 (+ 8238us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:07.819105 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:07.823488 (+ 4383us) server_negotiation.cc:407] Connection header received
0812 01:12:07.823748 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:07.823755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:07.823992 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:07.824292 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:07.827393 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.828622 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.831663 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:07.833173 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:07.833304 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:07.833538 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:07.834474 (+ 936us) server_negotiation.cc:299] Negotiation successful
0812 01:12:07.834667 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8025,"thread_start_us":7903,"threads_started":1}
I20260812 01:12:07.836593 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 759ec86c5a6b4be4934f7dd77806cd2f (DEFAULT_TABLE table=test-table [id=5c951bc8c2e345c9a15e9fef2e0ed4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:07.838303 1038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759ec86c5a6b4be4934f7dd77806cd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:07.838574 1316 tablet_service.cc:1467] Processing CreateTablet for tablet 759ec86c5a6b4be4934f7dd77806cd2f (DEFAULT_TABLE table=test-table [id=5c951bc8c2e345c9a15e9fef2e0ed4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:07.839802 1316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759ec86c5a6b4be4934f7dd77806cd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:07.839385 1566 tablet_service.cc:1467] Processing CreateTablet for tablet 759ec86c5a6b4be4934f7dd77806cd2f (DEFAULT_TABLE table=test-table [id=5c951bc8c2e345c9a15e9fef2e0ed4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:07.841136 1566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759ec86c5a6b4be4934f7dd77806cd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:07.854512 1758 tablet_bootstrap.cc:492] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: Bootstrap starting.
I20260812 01:12:07.859653 1758 tablet_bootstrap.cc:654] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:07.861457 1758 log.cc:824] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:07.861824 1763 tablet_bootstrap.cc:492] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: Bootstrap starting.
I20260812 01:12:07.863153 1761 tablet_bootstrap.cc:492] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: Bootstrap starting.
I20260812 01:12:07.866700 1763 tablet_bootstrap.cc:654] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:07.866833 1758 tablet_bootstrap.cc:492] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: No bootstrap required, opened a new log
I20260812 01:12:07.867228 1758 ts_tablet_manager.cc:1397] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260812 01:12:07.867992 1761 tablet_bootstrap.cc:654] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:07.868536 1763 log.cc:824] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:07.869410 1761 log.cc:824] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:07.872424 1763 tablet_bootstrap.cc:492] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: No bootstrap required, opened a new log
I20260812 01:12:07.872815 1763 ts_tablet_manager.cc:1397] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260812 01:12:07.873013 1761 tablet_bootstrap.cc:492] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: No bootstrap required, opened a new log
I20260812 01:12:07.873299 1761 ts_tablet_manager.cc:1397] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260812 01:12:07.878160 1758 raft_consensus.cc:348] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } } peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } }
I20260812 01:12:07.878711 1758 raft_consensus.cc:374] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:07.878832 1758 raft_consensus.cc:729] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1955a48e1490429484d27bf69dd64ab5, State: Initialized, Role: FOLLOWER
I20260812 01:12:07.879695 1758 consensus_queue.cc:260] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } } peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } }
I20260812 01:12:07.886826 1761 raft_consensus.cc:348] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } } peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } }
I20260812 01:12:07.887420 1761 raft_consensus.cc:374] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:07.887542 1761 raft_consensus.cc:729] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7fce201fee4ca3b83deab3cb7ac345, State: Initialized, Role: FOLLOWER
I20260812 01:12:07.888315 1758 ts_tablet_manager.cc:1428] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: Time spent starting tablet: real 0.021s user 0.004s sys 0.016s
I20260812 01:12:07.888254 1761 consensus_queue.cc:260] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } } peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } }
I20260812 01:12:07.888515 1763 raft_consensus.cc:348] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } } peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } }
I20260812 01:12:07.889206 1763 raft_consensus.cc:374] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:07.889369 1763 raft_consensus.cc:729] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec8159f12d9469aa0168790d61e4bda, State: Initialized, Role: FOLLOWER
I20260812 01:12:07.891995 1763 consensus_queue.cc:260] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } } peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } }
I20260812 01:12:07.894650 1435 heartbeater.cc:503] Master 127.20.151.62:33689 was elected leader, sending a full tablet report...
I20260812 01:12:07.903095 1761 ts_tablet_manager.cc:1428] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: Time spent starting tablet: real 0.029s user 0.020s sys 0.007s
I20260812 01:12:07.903712 1763 ts_tablet_manager.cc:1428] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: Time spent starting tablet: real 0.031s user 0.020s sys 0.007s
I20260812 01:12:07.904201 1688 heartbeater.cc:503] Master 127.20.151.62:33689 was elected leader, sending a full tablet report...
W20260812 01:12:07.906898 1165 tablet.cc:2369] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:12:07.955070 1691 tablet.cc:2369] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:12:08.027617 1771 raft_consensus.cc:482] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:12:08.028131 1771 raft_consensus.cc:504] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } } peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } }
I20260812 01:12:08.030351 1768 raft_consensus.cc:482] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:12:08.030841 1768 raft_consensus.cc:504] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } } peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } }
I20260812 01:12:08.030356 1771 leader_election.cc:288] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7fce201fee4ca3b83deab3cb7ac345 (127.20.151.1:33281), 1955a48e1490429484d27bf69dd64ab5 (127.20.151.2:34343)
I20260812 01:12:08.033102 1768 leader_election.cc:288] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7fce201fee4ca3b83deab3cb7ac345 (127.20.151.1:33281), eec8159f12d9469aa0168790d61e4bda (127.20.151.3:45441)
I20260812 01:12:08.043081 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.031542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:33281 (local address 127.20.151.3:53059)
0812 01:12:08.031691 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:08.031704 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:08.031828 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:08.032181 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:08.032186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:08.032213 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:08.032419 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:08.032430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.033891 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.033896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:08.038272 (+ 4376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:08.038290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.041860 (+ 3570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.041868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:08.042206 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.042255 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 01:12:08.042366 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 01:12:08.042561 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1819,"spinlock_wait_cycles":3456}
I20260812 01:12:08.044024 1706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.030583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:34343 (local address 127.20.151.3:54033)
0812 01:12:08.031650 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:08.031674 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:08.031828 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:08.032373 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:08.032377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:08.032398 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:08.032723 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:08.032732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.034441 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.034447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:08.036720 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:08.036740 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.042858 (+ 6118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.042865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:08.043327 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.043368 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 01:12:08.043465 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 01:12:08.043645 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260812 01:12:08.045403 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.031124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:54033 (local address 127.20.151.2:34343)
0812 01:12:08.031368 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:08.031374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:08.031949 (+ 575us) server_negotiation.cc:407] Connection header received
0812 01:12:08.032019 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:08.032022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:08.032077 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:08.032157 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:08.032917 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.034247 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.038571 (+ 4324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.042455 (+ 3884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.042627 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.042706 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:08.044288 (+ 1582us) server_negotiation.cc:299] Negotiation successful
0812 01:12:08.044437 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:12:08.046444 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.030888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:53059 (local address 127.20.151.1:33281)
0812 01:12:08.031133 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:08.031140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:08.031859 (+ 719us) server_negotiation.cc:407] Connection header received
0812 01:12:08.031957 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:08.031962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:08.032045 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:08.032147 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:08.032583 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.033734 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.039977 (+ 6243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.041625 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.041823 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.041914 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:08.045954 (+ 4040us) server_negotiation.cc:299] Negotiation successful
0812 01:12:08.046142 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 01:12:08.046586 1350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759ec86c5a6b4be4934f7dd77806cd2f" candidate_uuid: "eec8159f12d9469aa0168790d61e4bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1955a48e1490429484d27bf69dd64ab5" is_pre_election: true
I20260812 01:12:08.047254 1350 raft_consensus.cc:2454] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eec8159f12d9469aa0168790d61e4bda in term 0.
I20260812 01:12:08.047415 1072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759ec86c5a6b4be4934f7dd77806cd2f" candidate_uuid: "eec8159f12d9469aa0168790d61e4bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" is_pre_election: true
I20260812 01:12:08.047991 1072 raft_consensus.cc:2454] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eec8159f12d9469aa0168790d61e4bda in term 0.
I20260812 01:12:08.048398 1482 leader_election.cc:302] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1955a48e1490429484d27bf69dd64ab5, eec8159f12d9469aa0168790d61e4bda; no voters:
I20260812 01:12:08.048955 1771 raft_consensus.cc:2781] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:12:08.049140 1771 raft_consensus.cc:482] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:12:08.049333 1771 raft_consensus.cc:3037] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:08.051942 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.033956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:58575 (local address 127.20.151.3:45441)
0812 01:12:08.034203 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:08.034209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:08.036059 (+ 1850us) server_negotiation.cc:407] Connection header received
0812 01:12:08.036381 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:08.036388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:08.036465 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:08.036535 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:08.037771 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.039041 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.049047 (+ 10006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.050387 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.050531 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.050617 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:08.051505 (+ 888us) server_negotiation.cc:299] Negotiation successful
0812 01:12:08.051658 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 01:12:08.052942 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.033429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:45441 (local address 127.20.151.2:58575)
0812 01:12:08.035942 (+ 2513us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:08.035966 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:08.036208 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:08.036834 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:08.036839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:08.036868 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:08.037586 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:08.037600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.039243 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.039249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:08.048838 (+ 9589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:08.048857 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.050885 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.050891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:08.051289 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.051328 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 01:12:08.052144 (+ 816us) client_negotiation.cc:240] Negotiation successful
0812 01:12:08.052310 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2336,"mutex_wait_us":2,"thread_start_us":1935,"threads_started":1}
I20260812 01:12:08.053761 1771 raft_consensus.cc:504] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } } peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } }
I20260812 01:12:08.054260 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759ec86c5a6b4be4934f7dd77806cd2f" candidate_uuid: "1955a48e1490429484d27bf69dd64ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eec8159f12d9469aa0168790d61e4bda" is_pre_election: true
W20260812 01:12:08.054677 1439 tablet.cc:2369] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:12:08.054805 1600 raft_consensus.cc:2379] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1955a48e1490429484d27bf69dd64ab5 in current term 1: Already voted for candidate eec8159f12d9469aa0168790d61e4bda in this term.
I20260812 01:12:08.056344 1072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759ec86c5a6b4be4934f7dd77806cd2f" candidate_uuid: "eec8159f12d9469aa0168790d61e4bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7fce201fee4ca3b83deab3cb7ac345"
I20260812 01:12:08.056612 1072 raft_consensus.cc:3037] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:08.056907 1350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759ec86c5a6b4be4934f7dd77806cd2f" candidate_uuid: "eec8159f12d9469aa0168790d61e4bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1955a48e1490429484d27bf69dd64ab5"
I20260812 01:12:08.057159 1350 raft_consensus.cc:3037] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:08.057664 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.033905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:40299 (local address 127.20.151.1:33281)
0812 01:12:08.046715 (+ 12810us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:08.046721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:08.048041 (+ 1320us) server_negotiation.cc:407] Connection header received
0812 01:12:08.048120 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:08.048126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:08.048262 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:08.048354 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:08.050200 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.051418 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.054791 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.056168 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.056350 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.056428 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:08.057226 (+ 798us) server_negotiation.cc:299] Negotiation successful
0812 01:12:08.057372 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12691,"thread_start_us":11678,"threads_started":1}
I20260812 01:12:08.058372 1775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.034476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:33281 (local address 127.20.151.2:40299)
0812 01:12:08.047436 (+ 12960us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:08.047457 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:08.047626 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:08.049508 (+ 1882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:08.049514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:08.049545 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:08.049963 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:08.049979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.051624 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.051631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:08.054521 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:08.054541 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.056701 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.056710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:08.057020 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.057059 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 01:12:08.057827 (+ 768us) client_negotiation.cc:240] Negotiation successful
0812 01:12:08.057980 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12766,"thread_start_us":16624,"threads_started":1}
I20260812 01:12:08.059576 1771 leader_election.cc:288] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [CANDIDATE]: Term 1 election: Requested vote from peers dd7fce201fee4ca3b83deab3cb7ac345 (127.20.151.1:33281), 1955a48e1490429484d27bf69dd64ab5 (127.20.151.2:34343)
I20260812 01:12:08.059639 1070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759ec86c5a6b4be4934f7dd77806cd2f" candidate_uuid: "1955a48e1490429484d27bf69dd64ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" is_pre_election: true
I20260812 01:12:08.060736 1236 leader_election.cc:302] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1955a48e1490429484d27bf69dd64ab5; no voters: dd7fce201fee4ca3b83deab3cb7ac345, eec8159f12d9469aa0168790d61e4bda
I20260812 01:12:08.061144 1072 raft_consensus.cc:2454] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eec8159f12d9469aa0168790d61e4bda in term 1.
I20260812 01:12:08.061131 1350 raft_consensus.cc:2454] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eec8159f12d9469aa0168790d61e4bda in term 1.
I20260812 01:12:08.061549 1768 raft_consensus.cc:2726] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 01:12:08.061749 1489 leader_election.cc:302] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7fce201fee4ca3b83deab3cb7ac345, eec8159f12d9469aa0168790d61e4bda; no voters:
I20260812 01:12:08.062153 1771 raft_consensus.cc:2781] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:08.062918 1771 raft_consensus.cc:686] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 1 LEADER]: Becoming Leader. State: Replica: eec8159f12d9469aa0168790d61e4bda, State: Running, Role: LEADER
I20260812 01:12:08.063620 1771 consensus_queue.cc:237] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } } peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } }
I20260812 01:12:08.069664 31917 catalog_manager.cc:5495] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda reported cstate change: term changed from 0 to 1, leader changed from <none> to eec8159f12d9469aa0168790d61e4bda (127.20.151.3). New cstate: current_term: 1 leader_uuid: "eec8159f12d9469aa0168790d61e4bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } health_report { overall_health: UNKNOWN } } }
I20260812 01:12:08.136713 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.130595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.20.151.3:45441)
0812 01:12:08.130819 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:08.130826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:08.130946 (+ 120us) server_negotiation.cc:407] Connection header received
0812 01:12:08.131144 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:08.131148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:08.131243 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:08.131397 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:12:08.131829 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.132796 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.134311 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.134829 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.134943 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.135509 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:12:08.136010 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:12:08.136144 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:08.136257 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 01:12:08.136341 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:12:08.145773 1072 raft_consensus.cc:1261] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345 [term 1 FOLLOWER]: Refusing update from remote peer eec8159f12d9469aa0168790d61e4bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:08.145936 1350 raft_consensus.cc:1261] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 1 FOLLOWER]: Refusing update from remote peer eec8159f12d9469aa0168790d61e4bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:08.146775 1776 consensus_queue.cc:1035] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:08.147210 1771 consensus_queue.cc:1035] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:08.187472 1788 mvcc.cc:205] Tried to move back new op lower bound from 7317492236874571776 to 7317492236553748480. Current Snapshot: MvccSnapshot[applied={T|T < 7317492236874571776}]
I20260812 01:12:08.191959 31917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:08.192262 31917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:08.203915 31917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd3bef72f2d2436dbd3f6506788f0b68: Sending DeleteTablet for 3 replicas of tablet 759ec86c5a6b4be4934f7dd77806cd2f
I20260812 01:12:08.204252 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 759ec86c5a6b4be4934f7dd77806cd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:08 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:40516
I20260812 01:12:08.204864 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 759ec86c5a6b4be4934f7dd77806cd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:08 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:40516
I20260812 01:12:08.205291 1566 tablet_service.cc:1514] Processing DeleteTablet for tablet 759ec86c5a6b4be4934f7dd77806cd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:08 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:41558
I20260812 01:12:08.206493 1316 tablet_service.cc:1514] Processing DeleteTablet for tablet 759ec86c5a6b4be4934f7dd77806cd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:08 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:43984
I20260812 01:12:08.211066 1807 tablet_replica.cc:330] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: stopping tablet replica
I20260812 01:12:08.211669 1807 raft_consensus.cc:2229] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:12:08.212634 1807 raft_consensus.cc:2258] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:08.219971 1808 tablet_replica.cc:330] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: stopping tablet replica
I20260812 01:12:08.220790 1806 tablet_replica.cc:330] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: stopping tablet replica
I20260812 01:12:08.221024 1807 ts_tablet_manager.cc:1905] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:08.221213 1806 raft_consensus.cc:2229] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:08.221793 1806 raft_consensus.cc:2258] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:08.222586 1808 raft_consensus.cc:2229] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:08.223096 1808 raft_consensus.cc:2258] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:08.223120 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 31862
I20260812 01:12:08.224378 31971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:08.226500 1808 ts_tablet_manager.cc:1905] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:08.227429 1806 ts_tablet_manager.cc:1905] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:08.235311 1808 ts_tablet_manager.cc:1918] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:08.243816 1806 ts_tablet_manager.cc:1918] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:08.244164 1808 log.cc:1196] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/759ec86c5a6b4be4934f7dd77806cd2f
I20260812 01:12:08.244655 1722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.208709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.20.151.61:40015)
0812 01:12:08.208907 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:08.208913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:08.222826 (+ 13913us) server_negotiation.cc:407] Connection header received
0812 01:12:08.223138 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:08.223145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:08.223237 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:08.223342 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:08.228113 (+ 4771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.229441 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.231789 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.232596 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.232693 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.236834 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:08.236842 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:08.236847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:08.236891 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:08.240925 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:08.242007 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:08.242012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:08.242015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:08.242252 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:08.242695 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:08.242701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:08.242704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:08.243076 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:08.243373 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:08.244078 (+ 705us) server_negotiation.cc:299] Negotiation successful
0812 01:12:08.244320 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 01:12:08.245301 1808 ts_tablet_manager.cc:1939] T 759ec86c5a6b4be4934f7dd77806cd2f P 1955a48e1490429484d27bf69dd64ab5: Deleting consensus metadata
I20260812 01:12:08.245268 1807 ts_tablet_manager.cc:1918] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:08.245517 1807 log.cc:1196] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/759ec86c5a6b4be4934f7dd77806cd2f
I20260812 01:12:08.245523 1806 log.cc:1196] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/759ec86c5a6b4be4934f7dd77806cd2f
I20260812 01:12:08.246480 1807 ts_tablet_manager.cc:1939] T 759ec86c5a6b4be4934f7dd77806cd2f P eec8159f12d9469aa0168790d61e4bda: Deleting consensus metadata
I20260812 01:12:08.247108 1806 ts_tablet_manager.cc:1939] T 759ec86c5a6b4be4934f7dd77806cd2f P dd7fce201fee4ca3b83deab3cb7ac345: Deleting consensus metadata
I20260812 01:12:08.266208 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.222189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.20.151.60:34219)
0812 01:12:08.222417 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:08.222423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:08.247226 (+ 24803us) server_negotiation.cc:407] Connection header received
0812 01:12:08.247400 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:08.247407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:08.247499 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:08.247620 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:08.249792 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.250695 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.253093 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.253857 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.253997 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.258375 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:08.258383 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:08.258389 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:08.258438 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:08.262833 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:08.263736 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:08.263742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:08.263746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:08.263956 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:08.264393 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:08.264398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:08.264401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:08.264869 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:08.265124 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:08.265462 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 01:12:08.265810 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:12:08.361778 31886 catalog_manager.cc:4841] TS 1955a48e1490429484d27bf69dd64ab5 (127.20.151.2:34343): tablet 759ec86c5a6b4be4934f7dd77806cd2f (table test-table [id=5c951bc8c2e345c9a15e9fef2e0ed4ae]) successfully deleted
I20260812 01:12:08.362756 31891 catalog_manager.cc:4841] TS eec8159f12d9469aa0168790d61e4bda (127.20.151.3:45441): tablet 759ec86c5a6b4be4934f7dd77806cd2f (table test-table [id=5c951bc8c2e345c9a15e9fef2e0ed4ae]) successfully deleted
W20260812 01:12:08.363897 31895 catalog_manager.cc:4808] TS dd7fce201fee4ca3b83deab3cb7ac345 (127.20.151.1:33281): delete failed for tablet 759ec86c5a6b4be4934f7dd77806cd2f because the tablet was not found. No further retry: Not found: Tablet not found: 759ec86c5a6b4be4934f7dd77806cd2f
I20260812 01:12:08.364655 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 31862
I20260812 01:12:08.433291 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.405182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.20.151.60:34219)
0812 01:12:08.405654 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:08.405661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:08.416100 (+ 10439us) server_negotiation.cc:407] Connection header received
0812 01:12:08.416407 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:08.416414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:08.416503 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:08.416579 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:08.419007 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.420157 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.422693 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.423419 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.423515 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.426596 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:08.426603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:08.426608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:08.426652 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:08.430250 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:08.431020 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:08.431026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:08.431030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:08.431186 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:08.431642 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:08.431649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:08.431652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:08.432060 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:08.432308 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:08.432723 (+ 415us) server_negotiation.cc:299] Negotiation successful
0812 01:12:08.432952 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325}
I20260812 01:12:08.435794 1722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:08.420511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.20.151.61:40015)
0812 01:12:08.420715 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:08.420722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:08.420741 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:12:08.420798 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:08.420804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:08.420878 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:08.420974 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:08.423398 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.424476 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.426472 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:08.426938 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:08.427053 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:08.429643 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:08.429649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:08.429652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:08.429686 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:08.432820 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:08.433707 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:08.433713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:08.433717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:08.433875 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:08.434266 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:08.434271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:08.434274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:08.434626 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:08.434872 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:08.435156 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 01:12:08.435411 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:12:09.174238 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.173431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.2:43045)
0812 01:12:09.173921 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.174058 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":254,"threads_started":1}
W20260812 01:12:09.174633 1435 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:33689 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
I20260812 01:12:09.182603 1821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.181868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.3:54159)
0812 01:12:09.182287 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.182406 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":210,"threads_started":1}
W20260812 01:12:09.183030 1688 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:33689 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
I20260812 01:12:09.205574 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.204797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.1:47719)
0812 01:12:09.205232 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.205354 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":222,"threads_started":1}
W20260812 01:12:09.206004 1162 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:33689 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
I20260812 01:12:09.701035 1823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd3bef72f2d2436dbd3f6506788f0b68)
I20260812 01:12:09.701292 1823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } }
I20260812 01:12:09.702056 1825 raft_consensus.cc:482] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd3bef72f2d2436dbd3f6506788f0b68)
I20260812 01:12:09.702332 1823 leader_election.cc:288] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd3bef72f2d2436dbd3f6506788f0b68 (127.20.151.62:33689), a749f9fa01ae42248b34e414f8c225c0 (127.20.151.60:34219)
I20260812 01:12:09.702267 1825 raft_consensus.cc:504] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } }
I20260812 01:12:09.703111 1825 leader_election.cc:288] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd3bef72f2d2436dbd3f6506788f0b68 (127.20.151.62:33689), 9d65031b3e3745329fdb09dcf3882d61 (127.20.151.61:40015)
I20260812 01:12:09.704213 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.702700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:52946)
0812 01:12:09.703749 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.703920 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":920,"threads_started":1}
I20260812 01:12:09.705855 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.703253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:52960)
0812 01:12:09.705323 (+ 2070us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.705543 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1936,"thread_start_us":1721,"threads_started":1}
W20260812 01:12:09.706925 461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
W20260812 01:12:09.707195 807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
I20260812 01:12:09.712041 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.711359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:52962)
0812 01:12:09.711588 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.711792 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260812 01:12:09.712100 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.711457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:52970)
0812 01:12:09.711620 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.711861 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 01:12:09.712431 807 leader_election.cc:334] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd3bef72f2d2436dbd3f6506788f0b68 (127.20.151.62:33689): Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
W20260812 01:12:09.712428 461 leader_election.cc:334] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd3bef72f2d2436dbd3f6506788f0b68 (127.20.151.62:33689): Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
I20260812 01:12:09.715185 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.705383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:40015 (local address 127.0.0.1:59128)
0812 01:12:09.706161 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.706180 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:09.706315 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:09.707785 (+ 1470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:09.707791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:09.707822 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:09.708088 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.708100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.710944 (+ 2844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.710949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.713041 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.713055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.714618 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.714623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.714756 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.714781 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:12:09.714875 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 01:12:09.714977 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":1467,"threads_started":1}
I20260812 01:12:09.715646 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.705743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.20.151.61:40015)
0812 01:12:09.707432 (+ 1689us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:09.707439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:09.707470 (+ 31us) server_negotiation.cc:407] Connection header received
0812 01:12:09.707533 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:09.707537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:09.707612 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:09.708239 (+ 627us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:09.708512 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.710771 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.713205 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.714454 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.714597 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.714693 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:09.714908 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 01:12:09.715047 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":1575,"thread_start_us":1515,"threads_started":1}
I20260812 01:12:09.716336 497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a749f9fa01ae42248b34e414f8c225c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d65031b3e3745329fdb09dcf3882d61" is_pre_election: true
I20260812 01:12:09.716648 497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a749f9fa01ae42248b34e414f8c225c0 in term 1.
I20260812 01:12:09.717118 807 leader_election.cc:302] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d65031b3e3745329fdb09dcf3882d61, a749f9fa01ae42248b34e414f8c225c0; no voters: cd3bef72f2d2436dbd3f6506788f0b68
I20260812 01:12:09.717439 1825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 01:12:09.717643 1825 raft_consensus.cc:482] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd3bef72f2d2436dbd3f6506788f0b68)
I20260812 01:12:09.717751 1825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 1 FOLLOWER]: Advancing to term 2
I20260812 01:12:09.717936 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.702767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:34219 (local address 127.0.0.1:46814)
0812 01:12:09.704566 (+ 1799us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.704590 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:09.704747 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:09.711441 (+ 6694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:09.711455 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:09.711485 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:09.711829 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.711843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.713623 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.713629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.715606 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.715626 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.717127 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.717133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.717322 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.717354 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:12:09.717499 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 01:12:09.717622 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1655,"mutex_wait_us":3,"thread_start_us":717,"threads_started":1}
I20260812 01:12:09.718043 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.703248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.20.151.60:34219)
0812 01:12:09.710476 (+ 7228us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:09.710483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:09.710506 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:12:09.710573 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:09.710582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:09.710654 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:09.711983 (+ 1329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:09.712258 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.713458 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.715902 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.716954 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.717072 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.717166 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:09.717530 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 01:12:09.717674 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7052,"thread_start_us":6874,"threads_started":1}
I20260812 01:12:09.718636 850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d65031b3e3745329fdb09dcf3882d61" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a749f9fa01ae42248b34e414f8c225c0" is_pre_election: true
I20260812 01:12:09.720293 1825 raft_consensus.cc:504] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } }
I20260812 01:12:09.720851 850 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9d65031b3e3745329fdb09dcf3882d61 in current term 2: Already voted for candidate a749f9fa01ae42248b34e414f8c225c0 in this term.
I20260812 01:12:09.721019 1825 leader_election.cc:288] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [CANDIDATE]: Term 2 election: Requested vote from peers cd3bef72f2d2436dbd3f6506788f0b68 (127.20.151.62:33689), 9d65031b3e3745329fdb09dcf3882d61 (127.20.151.61:40015)
I20260812 01:12:09.721587 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.721167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:52980)
0812 01:12:09.721309 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.721402 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 01:12:09.721844 497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a749f9fa01ae42248b34e414f8c225c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d65031b3e3745329fdb09dcf3882d61"
I20260812 01:12:09.721968 458 leader_election.cc:302] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d65031b3e3745329fdb09dcf3882d61; no voters: a749f9fa01ae42248b34e414f8c225c0, cd3bef72f2d2436dbd3f6506788f0b68
I20260812 01:12:09.722084 497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 1 FOLLOWER]: Advancing to term 2
I20260812 01:12:09.722674 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.722248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:52988)
0812 01:12:09.722449 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.722544 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260812 01:12:09.722895 807 leader_election.cc:334] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cd3bef72f2d2436dbd3f6506788f0b68 (127.20.151.62:33689): Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
I20260812 01:12:09.724792 497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a749f9fa01ae42248b34e414f8c225c0 in term 2.
I20260812 01:12:09.724982 1823 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 01:12:09.725263 807 leader_election.cc:302] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d65031b3e3745329fdb09dcf3882d61, a749f9fa01ae42248b34e414f8c225c0; no voters: cd3bef72f2d2436dbd3f6506788f0b68
I20260812 01:12:09.725606 1825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 01:12:09.726086 1825 raft_consensus.cc:686] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [term 2 LEADER]: Becoming Leader. State: Replica: a749f9fa01ae42248b34e414f8c225c0, State: Running, Role: LEADER
I20260812 01:12:09.726511 1825 consensus_queue.cc:237] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } }
I20260812 01:12:09.728282 1847 sys_catalog.cc:455] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a749f9fa01ae42248b34e414f8c225c0. Latest consensus state: current_term: 2 leader_uuid: "a749f9fa01ae42248b34e414f8c225c0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:09.728678 1847 sys_catalog.cc:458] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:09.729353 1851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:12:09.733114 1851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:12:09.733896 1851 catalog_manager.cc:1259] Loaded cluster ID: 1c08e6e0a00e4060aacdd304bfe6627b
I20260812 01:12:09.733981 1851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:12:09.735116 1851 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:12:09.736001 1851 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Loaded TSK: 0
I20260812 01:12:09.736829 1851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:12:09.782543 828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:12:09.789840 828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:12:09.794167 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.793710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:53008)
0812 01:12:09.793867 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.793982 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 01:12:09.794570 497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [term 2 FOLLOWER]: Refusing update from remote peer a749f9fa01ae42248b34e414f8c225c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260812 01:12:09.795130 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.794775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:53012)
0812 01:12:09.794905 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.794988 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 01:12:09.795305 807 consensus_peers.cc:487] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 -> Peer cd3bef72f2d2436dbd3f6506788f0b68 (127.20.151.62:33689): Couldn't send request to peer cd3bef72f2d2436dbd3f6506788f0b68. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 01:12:09.795387 1847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 01:12:09.798449 1823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a749f9fa01ae42248b34e414f8c225c0. Latest consensus state: current_term: 2 leader_uuid: "a749f9fa01ae42248b34e414f8c225c0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:09.798739 1823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:12:09.800274 1847 sys_catalog.cc:455] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a749f9fa01ae42248b34e414f8c225c0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:09.800660 1847 sys_catalog.cc:458] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:09.801102 1823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a749f9fa01ae42248b34e414f8c225c0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "cd3bef72f2d2436dbd3f6506788f0b68" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 33689 } } peers { permanent_uuid: "9d65031b3e3745329fdb09dcf3882d61" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 40015 } attrs { promote: false } } peers { permanent_uuid: "a749f9fa01ae42248b34e414f8c225c0" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 34219 } attrs { promote: false } } }
I20260812 01:12:09.801406 1823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d65031b3e3745329fdb09dcf3882d61 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:12:09.817174 1872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.809472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:33281 (local address 127.0.0.1:40530)
0812 01:12:09.810410 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.810429 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:09.810545 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:09.810860 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:09.810865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:09.810888 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:09.811369 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.811385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.813129 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.813133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.814796 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.814809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.816221 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.816228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.816449 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.816491 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 01:12:09.816577 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 01:12:09.816703 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":63,"spinlock_wait_cycles":12032,"thread_start_us":780,"threads_started":1}
I20260812 01:12:09.817744 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.810104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.20.151.2:34343)
0812 01:12:09.810632 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:09.810639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:09.810656 (+ 17us) server_negotiation.cc:407] Connection header received
0812 01:12:09.810734 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:09.810740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:09.810818 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:09.811121 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:09.811654 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.812703 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.815515 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.816714 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.816895 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.817013 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:09.817315 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 01:12:09.817456 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":288,"threads_started":1}
I20260812 01:12:09.817739 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.809226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:34343 (local address 127.0.0.1:43988)
0812 01:12:09.809733 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.809749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:09.809856 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:09.811006 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:09.811012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:09.811058 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:09.811512 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.811524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.812846 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.812851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.815260 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.815300 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.816974 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.816981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.817169 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.817192 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 01:12:09.817272 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 01:12:09.817404 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":128,"spinlock_wait_cycles":768}
I20260812 01:12:09.818332 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.809642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.20.151.1:33281)
0812 01:12:09.810187 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:09.810194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:09.810496 (+ 302us) server_negotiation.cc:407] Connection header received
0812 01:12:09.810688 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:09.810692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:09.810754 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:09.810833 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:09.811593 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.813000 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.814948 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.815979 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.817718 (+ 1739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.817828 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:09.817880 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 01:12:09.818016 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":259,"threads_started":1}
I20260812 01:12:09.818720 1316 tablet_service.cc:1467] Processing CreateTablet for tablet e70906a31d3a4922a83a804857bdd1a6 (DEFAULT_TABLE table=test-table [id=550baf64640e4b1fa674ce5f12c1493c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:09.819134 1316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70906a31d3a4922a83a804857bdd1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:09.819036 1038 tablet_service.cc:1467] Processing CreateTablet for tablet e70906a31d3a4922a83a804857bdd1a6 (DEFAULT_TABLE table=test-table [id=550baf64640e4b1fa674ce5f12c1493c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:09.819506 1038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70906a31d3a4922a83a804857bdd1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:09.823066 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.808718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:45441 (local address 127.0.0.1:41576)
0812 01:12:09.808899 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.808919 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:09.809026 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:09.816261 (+ 7235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:09.816265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:09.816291 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:09.816547 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.816559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.817984 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.817990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.819625 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.819639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.822322 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.822329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.822500 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.822532 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:12:09.822633 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 01:12:09.822784 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":3}
I20260812 01:12:09.823153 1867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.808857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.20.151.3:45441)
0812 01:12:09.815866 (+ 7009us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:09.815872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:09.815891 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:12:09.815962 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:09.815966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:09.816111 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:09.816684 (+ 573us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:09.816884 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.817833 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.820910 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.822156 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.822287 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.822385 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:09.822669 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 01:12:09.822815 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6875,"thread_start_us":6597,"threads_started":1}
I20260812 01:12:09.823838 1566 tablet_service.cc:1467] Processing CreateTablet for tablet e70906a31d3a4922a83a804857bdd1a6 (DEFAULT_TABLE table=test-table [id=550baf64640e4b1fa674ce5f12c1493c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:09.824239 1566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70906a31d3a4922a83a804857bdd1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:09.825170 1876 tablet_bootstrap.cc:492] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345: Bootstrap starting.
I20260812 01:12:09.827107 1876 tablet_bootstrap.cc:654] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:09.829412 1876 tablet_bootstrap.cc:492] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345: No bootstrap required, opened a new log
I20260812 01:12:09.829654 1876 ts_tablet_manager.cc:1397] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 01:12:09.829790 1879 tablet_bootstrap.cc:492] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5: Bootstrap starting.
I20260812 01:12:09.830746 1880 tablet_bootstrap.cc:492] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda: Bootstrap starting.
I20260812 01:12:09.831084 1876 raft_consensus.cc:348] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } } peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } }
I20260812 01:12:09.831499 1876 raft_consensus.cc:374] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:09.831605 1879 tablet_bootstrap.cc:654] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:09.831669 1876 raft_consensus.cc:729] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7fce201fee4ca3b83deab3cb7ac345, State: Initialized, Role: FOLLOWER
I20260812 01:12:09.831962 1876 consensus_queue.cc:260] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } } peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } }
I20260812 01:12:09.832657 1880 tablet_bootstrap.cc:654] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:09.832772 1876 ts_tablet_manager.cc:1428] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 01:12:09.834173 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.833149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.1:41389)
0812 01:12:09.833703 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.833881 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":602,"threads_started":1}
I20260812 01:12:09.836133 1161 heartbeater.cc:503] Master 127.20.151.60:34219 was elected leader, sending a full tablet report...
I20260812 01:12:09.839118 1881 raft_consensus.cc:482] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:12:09.839442 1881 raft_consensus.cc:504] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } } peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } }
I20260812 01:12:09.841267 1880 tablet_bootstrap.cc:492] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda: No bootstrap required, opened a new log
I20260812 01:12:09.841439 1880 ts_tablet_manager.cc:1397] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260812 01:12:09.842242 1881 leader_election.cc:288] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eec8159f12d9469aa0168790d61e4bda (127.20.151.3:45441), 1955a48e1490429484d27bf69dd64ab5 (127.20.151.2:34343)
I20260812 01:12:09.842438 1880 raft_consensus.cc:348] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } } peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } }
I20260812 01:12:09.842700 1880 raft_consensus.cc:374] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:09.842804 1880 raft_consensus.cc:729] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec8159f12d9469aa0168790d61e4bda, State: Initialized, Role: FOLLOWER
I20260812 01:12:09.843073 1880 consensus_queue.cc:260] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } } peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } }
I20260812 01:12:09.843820 1879 tablet_bootstrap.cc:492] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5: No bootstrap required, opened a new log
I20260812 01:12:09.843995 1879 ts_tablet_manager.cc:1397] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260812 01:12:09.844877 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.834853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.1:58245)
0812 01:12:09.838621 (+ 3768us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.838831 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3649,"threads_started":1}
I20260812 01:12:09.844848 1879 raft_consensus.cc:348] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } } peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } }
I20260812 01:12:09.845113 1879 raft_consensus.cc:374] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:09.845211 1879 raft_consensus.cc:729] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1955a48e1490429484d27bf69dd64ab5, State: Initialized, Role: FOLLOWER
I20260812 01:12:09.845475 1879 consensus_queue.cc:260] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } } peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } }
I20260812 01:12:09.847806 1880 ts_tablet_manager.cc:1428] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260812 01:12:09.849354 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.841881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:44639 (local address 127.20.151.2:34343)
0812 01:12:09.842075 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:09.842081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:09.842115 (+ 34us) server_negotiation.cc:407] Connection header received
0812 01:12:09.842184 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:09.842189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:09.842262 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:09.842353 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:09.842974 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.844074 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.846878 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.848083 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.848217 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.848294 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:09.848932 (+ 638us) server_negotiation.cc:299] Negotiation successful
0812 01:12:09.849075 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 01:12:09.849534 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.848536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.3:38549)
0812 01:12:09.849068 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.849283 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"threads_started":1}
I20260812 01:12:09.850106 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.841247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:34343 (local address 127.20.151.1:44639)
0812 01:12:09.841594 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.841614 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:09.841955 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:09.842453 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:09.842472 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:09.842502 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:09.842783 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.842794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.844248 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.844253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.846694 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.846709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.848493 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.848500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.848710 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.848748 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 01:12:09.849598 (+ 850us) client_negotiation.cc:240] Negotiation successful
0812 01:12:09.849758 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":329}
I20260812 01:12:09.850733 1350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70906a31d3a4922a83a804857bdd1a6" candidate_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1955a48e1490429484d27bf69dd64ab5" is_pre_election: true
I20260812 01:12:09.850988 1350 raft_consensus.cc:2454] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7fce201fee4ca3b83deab3cb7ac345 in term 0.
I20260812 01:12:09.851606 954 leader_election.cc:302] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1955a48e1490429484d27bf69dd64ab5, dd7fce201fee4ca3b83deab3cb7ac345; no voters:
I20260812 01:12:09.852028 1881 raft_consensus.cc:2781] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:12:09.852057 1686 heartbeater.cc:503] Master 127.20.151.60:34219 was elected leader, sending a full tablet report...
I20260812 01:12:09.852138 1881 raft_consensus.cc:482] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:12:09.852234 1881 raft_consensus.cc:3037] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:09.852730 1879 ts_tablet_manager.cc:1428] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5: Time spent starting tablet: real 0.009s user 0.000s sys 0.006s
I20260812 01:12:09.853807 1867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.841618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:46499 (local address 127.20.151.3:45441)
0812 01:12:09.841845 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:09.841852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:09.843917 (+ 2065us) server_negotiation.cc:407] Connection header received
0812 01:12:09.844284 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:09.844289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:09.844365 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:09.844455 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:09.845511 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.846583 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.851263 (+ 4680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.852527 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.852643 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.852709 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:09.853446 (+ 737us) server_negotiation.cc:299] Negotiation successful
0812 01:12:09.853576 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 01:12:09.854624 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.841277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:45441 (local address 127.20.151.1:46499)
0812 01:12:09.843821 (+ 2544us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.843839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:09.844016 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:09.844607 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:09.844612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:09.844636 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:09.845254 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.845310 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.848147 (+ 2837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.848151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.850998 (+ 2847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:09.851014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.853057 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.853063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:09.853244 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.853281 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 01:12:09.854096 (+ 815us) client_negotiation.cc:240] Negotiation successful
0812 01:12:09.854245 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2435,"mutex_wait_us":3,"thread_start_us":2307,"threads_started":1}
I20260812 01:12:09.855389 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70906a31d3a4922a83a804857bdd1a6" candidate_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eec8159f12d9469aa0168790d61e4bda" is_pre_election: true
I20260812 01:12:09.855619 1434 heartbeater.cc:503] Master 127.20.151.60:34219 was elected leader, sending a full tablet report...
I20260812 01:12:09.855640 1600 raft_consensus.cc:2454] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7fce201fee4ca3b83deab3cb7ac345 in term 0.
I20260812 01:12:09.856122 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.851535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.3:43471)
0812 01:12:09.855878 (+ 4343us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.855978 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4218}
I20260812 01:12:09.858937 1881 raft_consensus.cc:504] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } } peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } }
I20260812 01:12:09.859938 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70906a31d3a4922a83a804857bdd1a6" candidate_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eec8159f12d9469aa0168790d61e4bda"
I20260812 01:12:09.860029 1350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70906a31d3a4922a83a804857bdd1a6" candidate_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1955a48e1490429484d27bf69dd64ab5"
I20260812 01:12:09.860162 1600 raft_consensus.cc:3037] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:09.860190 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.853215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.2:39021)
0812 01:12:09.859774 (+ 6559us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.859911 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6434,"thread_start_us":6320,"threads_started":1}
I20260812 01:12:09.860262 1350 raft_consensus.cc:3037] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:09.861366 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.860982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.2:51305)
0812 01:12:09.861138 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.861223 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 01:12:09.863169 1350 raft_consensus.cc:2454] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7fce201fee4ca3b83deab3cb7ac345 in term 1.
I20260812 01:12:09.863169 1600 raft_consensus.cc:2454] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7fce201fee4ca3b83deab3cb7ac345 in term 1.
I20260812 01:12:09.863505 1881 leader_election.cc:288] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [CANDIDATE]: Term 1 election: Requested vote from peers eec8159f12d9469aa0168790d61e4bda (127.20.151.3:45441), 1955a48e1490429484d27bf69dd64ab5 (127.20.151.2:34343)
I20260812 01:12:09.863644 954 leader_election.cc:302] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1955a48e1490429484d27bf69dd64ab5, dd7fce201fee4ca3b83deab3cb7ac345; no voters:
I20260812 01:12:09.863919 1881 raft_consensus.cc:2781] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:09.864352 1881 raft_consensus.cc:686] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 1 LEADER]: Becoming Leader. State: Replica: dd7fce201fee4ca3b83deab3cb7ac345, State: Running, Role: LEADER
I20260812 01:12:09.864773 1881 consensus_queue.cc:237] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } } peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } }
I20260812 01:12:09.865254 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.864892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.1:34881)
0812 01:12:09.865032 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.865117 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 01:12:09.868388 828 catalog_manager.cc:5495] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7fce201fee4ca3b83deab3cb7ac345 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7fce201fee4ca3b83deab3cb7ac345" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 33281 } health_report { overall_health: HEALTHY } } }
I20260812 01:12:09.898976 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.894027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.20.151.1:33281)
0812 01:12:09.894229 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:09.894234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:09.894252 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:12:09.894349 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:09.894352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:09.894430 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:09.894552 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:12:09.894827 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.895598 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.896876 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.897352 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.897445 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.897854 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:12:09.898449 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:12:09.898587 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:09.898625 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 01:12:09.898694 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 01:12:09.903115 1350 raft_consensus.cc:1261] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5 [term 1 FOLLOWER]: Refusing update from remote peer dd7fce201fee4ca3b83deab3cb7ac345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:09.903378 1600 raft_consensus.cc:1261] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda [term 1 FOLLOWER]: Refusing update from remote peer dd7fce201fee4ca3b83deab3cb7ac345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:09.903901 1899 consensus_queue.cc:1035] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1955a48e1490429484d27bf69dd64ab5" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:09.904304 1881 consensus_queue.cc:1035] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eec8159f12d9469aa0168790d61e4bda" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:09.906739 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.906259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.3:38127)
0812 01:12:09.906426 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.906526 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 01:12:09.907716 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.907227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.2:34801)
0812 01:12:09.907399 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.907490 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 01:12:09.913455 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.912955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.3:41377)
0812 01:12:09.913134 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.913255 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 01:12:09.915508 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.913872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.1:48391)
0812 01:12:09.915207 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.915336 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1188}
I20260812 01:12:09.919404 1910 mvcc.cc:205] Tried to move back new op lower bound from 7317492244076322816 to 7317492243930370048. Current Snapshot: MvccSnapshot[applied={T|T < 7317492244076322816 or (T in {7317492244076322816})}]
I20260812 01:12:09.925302 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.924697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.1:52727)
0812 01:12:09.924953 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.925090 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 01:12:09.928118 828 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:09.929929 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.929322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.20.151.2:41625)
0812 01:12:09.929648 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:09.929760 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
I20260812 01:12:09.928471 828 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:09.937114 828 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0: Sending DeleteTablet for 3 replicas of tablet e70906a31d3a4922a83a804857bdd1a6
I20260812 01:12:09.938090 1566 tablet_service.cc:1514] Processing DeleteTablet for tablet e70906a31d3a4922a83a804857bdd1a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:09 UTC) from {username='oryx', principal='oryx/127.20.151.60@KRBTEST.COM'} at 127.0.0.1:41576
I20260812 01:12:09.938129 1316 tablet_service.cc:1514] Processing DeleteTablet for tablet e70906a31d3a4922a83a804857bdd1a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:09 UTC) from {username='oryx', principal='oryx/127.20.151.60@KRBTEST.COM'} at 127.0.0.1:43988
I20260812 01:12:09.938265 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet e70906a31d3a4922a83a804857bdd1a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:09 UTC) from {username='oryx', principal='oryx/127.20.151.60@KRBTEST.COM'} at 127.0.0.1:40530
I20260812 01:12:09.938851 1926 tablet_replica.cc:330] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda: stopping tablet replica
I20260812 01:12:09.939031 1926 raft_consensus.cc:2229] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:09.939350 1926 raft_consensus.cc:2258] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:09.939975 1925 tablet_replica.cc:330] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5: stopping tablet replica
I20260812 01:12:09.940229 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 801
I20260812 01:12:09.940572 1925 raft_consensus.cc:2229] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:09.940879 1925 raft_consensus.cc:2258] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:09.943372 1925 ts_tablet_manager.cc:1905] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:09.950831 1925 ts_tablet_manager.cc:1918] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:09.950994 1925 log.cc:1196] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/e70906a31d3a4922a83a804857bdd1a6
I20260812 01:12:09.951493 1925 ts_tablet_manager.cc:1939] T e70906a31d3a4922a83a804857bdd1a6 P 1955a48e1490429484d27bf69dd64ab5: Deleting consensus metadata
I20260812 01:12:09.952147 1929 tablet_replica.cc:330] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345: stopping tablet replica
I20260812 01:12:09.952363 1929 raft_consensus.cc:2229] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:12:09.952798 805 catalog_manager.cc:4841] TS 1955a48e1490429484d27bf69dd64ab5 (127.20.151.2:34343): tablet e70906a31d3a4922a83a804857bdd1a6 (table test-table [id=550baf64640e4b1fa674ce5f12c1493c]) successfully deleted
I20260812 01:12:09.942032 1926 ts_tablet_manager.cc:1905] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:09.953042 1929 raft_consensus.cc:2258] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:09.954332 1929 ts_tablet_manager.cc:1905] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:09.960181 1926 ts_tablet_manager.cc:1918] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:09.960351 1926 log.cc:1196] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/e70906a31d3a4922a83a804857bdd1a6
I20260812 01:12:09.960852 1926 ts_tablet_manager.cc:1939] T e70906a31d3a4922a83a804857bdd1a6 P eec8159f12d9469aa0168790d61e4bda: Deleting consensus metadata
I20260812 01:12:09.961567 1929 ts_tablet_manager.cc:1918] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:09.961726 1929 log.cc:1196] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/e70906a31d3a4922a83a804857bdd1a6
I20260812 01:12:09.962136 808 catalog_manager.cc:4841] TS eec8159f12d9469aa0168790d61e4bda (127.20.151.3:45441): tablet e70906a31d3a4922a83a804857bdd1a6 (table test-table [id=550baf64640e4b1fa674ce5f12c1493c]) successfully deleted
I20260812 01:12:09.962214 1929 ts_tablet_manager.cc:1939] T e70906a31d3a4922a83a804857bdd1a6 P dd7fce201fee4ca3b83deab3cb7ac345: Deleting consensus metadata
I20260812 01:12:09.963495 811 catalog_manager.cc:4841] TS dd7fce201fee4ca3b83deab3cb7ac345 (127.20.151.1:33281): tablet e70906a31d3a4922a83a804857bdd1a6 (table test-table [id=550baf64640e4b1fa674ce5f12c1493c]) successfully deleted
I20260812 01:12:09.969375 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:09.941893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.20.151.1:33281)
0812 01:12:09.942085 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:09.942091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:09.952146 (+ 10055us) server_negotiation.cc:407] Connection header received
0812 01:12:09.952804 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:09.952810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:09.952934 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:09.953018 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:09.955232 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.956519 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.958301 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:09.958979 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:09.959103 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:09.963037 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:09.963052 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:09.963073 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:09.963112 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:09.966373 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:09.967112 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:09.967121 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:09.967123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:09.967378 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:09.967757 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:09.967763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:09.967766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:09.968161 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:09.968461 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:09.968757 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 01:12:09.968992 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 01:12:09.970111 1153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:10.087774 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 801
I20260812 01:12:10.116348 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 1187
I20260812 01:12:10.130271 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:10.117156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.20.151.2:34343)
0812 01:12:10.117354 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:10.117358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:10.117377 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:12:10.117532 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:10.117536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:10.117605 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:10.117703 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:10.119012 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:10.119952 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:10.121263 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:10.121824 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:10.121954 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:10.124608 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:10.124627 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:10.124644 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:10.124687 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:10.127699 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:10.128318 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:10.128325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:10.128327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:10.128475 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:10.128803 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:10.128806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:10.128808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:10.129200 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:10.129440 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:10.129719 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 01:12:10.129919 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 01:12:10.130913 1425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:10.228120 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 1187
I20260812 01:12:10.237555 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:10.235941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:53014)
0812 01:12:10.237174 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:10.237311 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1082}
I20260812 01:12:10.239763 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:10.239370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:33689 (local address 127.0.0.1:53028)
0812 01:12:10.239517 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:10.239617 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 01:12:10.260754 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 1453
I20260812 01:12:10.274824 1867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:10.261534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.20.151.3:45441)
0812 01:12:10.261762 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:10.261769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:10.261835 (+ 66us) server_negotiation.cc:407] Connection header received
0812 01:12:10.262032 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:10.262038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:10.262124 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:10.262235 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:10.263689 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:10.264444 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:10.265666 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:10.266075 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:10.266188 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:10.268914 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:10.268933 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:10.268952 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:10.268997 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:10.271998 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:10.272735 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:10.272743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:10.272745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:10.272881 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:10.273261 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:10.273265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:10.273267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:10.273656 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:10.273978 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:10.274265 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 01:12:10.274495 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 01:12:10.275648 1679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:10.397790 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 1453
I20260812 01:12:10.428623 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 32706
I20260812 01:12:10.429812 533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:10.551792 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 32706
I20260812 01:12:10.581418 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 424
I20260812 01:12:10.582159 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:10.581689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:40015 (local address 127.0.0.1:59136)
0812 01:12:10.581888 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:10.581970 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 01:12:10.582383 807 consensus_peers.cc:487] T 00000000000000000000000000000000 P a749f9fa01ae42248b34e414f8c225c0 -> Peer 9d65031b3e3745329fdb09dcf3882d61 (127.20.151.61:40015): Couldn't send request to peer 9d65031b3e3745329fdb09dcf3882d61. Status: Network error: Client connection negotiation failed: client connection to 127.20.151.61:40015: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 01:12:10.582536 882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:10.709362 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 424
2026-08-12T01:12:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10553 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:12:10 dist-test-slave-21b9 krb5kdc[1949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:12:10 dist-test-slave-21b9 krb5kdc[1949](info): set up 2 sockets
Aug 12 01:12:10 dist-test-slave-21b9 krb5kdc[1949](info): commencing operation
krb5kdc: starting...
W20260812 01:12:11.994297 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.004s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:12:12 dist-test-slave-21b9 krb5kdc[1949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:12:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.62.keytab.
Entry for principal oryx/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.62.keytab.
I20260812 01:12:12.295580 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:34279
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.62.keytab
--principal=oryx/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:34279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:12.640064 1979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:12.640353 1979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:12.640481 1979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:12.647814 1979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:12:12.647926 1979 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 01:12:12.647992 1979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:12.648080 1979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:12.648137 1979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:12:12.648203 1979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:12:12.658617 1979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:34279
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.62.keytab
--principal=oryx/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:12.659942 1979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:12.661253 1979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:12.673322 1988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:12.673281 1993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:12.673252 1991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:12.674844 1979 server_base.cc:1034] running on GCE node
I20260812 01:12:12.675714 1979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:12.677114 1979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:12.678333 1979 hybrid_clock.cc:648] HybridClock initialized: now 1786497132678252 us; error 47 us; skew 500 ppm
I20260812 01:12:12.678720 1979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:12 dist-test-slave-21b9 krb5kdc[1949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497132, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:12.682559 1979 init.cc:377] Logged in from keytab as oryx/127.20.151.62@KRBTEST.COM (short username oryx)
I20260812 01:12:12.684067 1979 webserver.cc:460] Webserver started at http://127.20.151.62:34277/ using document root <none> and password file <none>
I20260812 01:12:12.684772 1979 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:12.684861 1979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:12.685132 1979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:12.688130 1979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "7e7d83789b294c80b971f67fe4151dd8"
format_stamp: "Formatted at 2026-08-12 01:12:12 on dist-test-slave-21b9"
server_key: "17644934b3e90b0929a732e08a620c98"
server_key_iv: "2c327e12af666ffac1b7962bbaa7b63e"
server_key_version: "encryptionkey@0"
I20260812 01:12:12.688921 1979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "7e7d83789b294c80b971f67fe4151dd8"
format_stamp: "Formatted at 2026-08-12 01:12:12 on dist-test-slave-21b9"
server_key: "17644934b3e90b0929a732e08a620c98"
server_key_iv: "2c327e12af666ffac1b7962bbaa7b63e"
server_key_version: "encryptionkey@0"
I20260812 01:12:12.694046 1979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:12:12.697520 2012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:12.698792 1979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:12:12.698954 1979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "7e7d83789b294c80b971f67fe4151dd8"
format_stamp: "Formatted at 2026-08-12 01:12:12 on dist-test-slave-21b9"
server_key: "17644934b3e90b0929a732e08a620c98"
server_key_iv: "2c327e12af666ffac1b7962bbaa7b63e"
server_key_version: "encryptionkey@0"
I20260812 01:12:12.699172 1979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:12.727634 1979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:12.731956 1979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:12.732244 1979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:12.754354 1979 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:34279
I20260812 01:12:12.754371 2126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:34279 every 8 connection(s)
I20260812 01:12:12.755802 1979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:12:12.759728 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:12.765492 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 1979
I20260812 01:12:12.765724 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:12:12.766390 21084 external_mini_cluster.cc:1428] Setting key 3d4e631e99c32123038d18caa04826b2
I20260812 01:12:12.770653 2128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8: Bootstrap starting.
I20260812 01:12:12.774524 2128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:12.775743 2128 log.cc:824] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:12.779978 2128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8: No bootstrap required, opened a new log
Aug 12 01:12:12 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:12.788290 2128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7d83789b294c80b971f67fe4151dd8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34279 } }
I20260812 01:12:12.788807 2128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:12.788926 2128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7d83789b294c80b971f67fe4151dd8, State: Initialized, Role: FOLLOWER
I20260812 01:12:12.789577 2128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7d83789b294c80b971f67fe4151dd8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34279 } }
I20260812 01:12:12.789831 2128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:12:12.789980 2128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:12:12.790158 2128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:12.792582 2128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7d83789b294c80b971f67fe4151dd8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34279 } }
I20260812 01:12:12.793141 2128 leader_election.cc:302] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e7d83789b294c80b971f67fe4151dd8; no voters:
I20260812 01:12:12.793821 2128 leader_election.cc:288] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:12:12.793946 2136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:12.795256 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:12.768551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.20.151.62:34279)
0812 01:12:12.769442 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:12.769454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:12.769497 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:12:12.771251 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:12.771299 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:12.771692 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:12.771999 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:12.773914 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:12.775338 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:12.776935 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:12.778514 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:12.778723 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:12.786670 (+ 7947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:12.786680 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:12.786705 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:12.786759 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:12.790334 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:12.791156 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:12.791167 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:12.791176 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:12.791442 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:12.791867 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:12.791872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:12.791876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:12.792506 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:12.792800 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:12.793329 (+ 529us) server_negotiation.cc:299] Negotiation successful
0812 01:12:12.793858 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":388,"threads_started":1}
I20260812 01:12:12.796365 2128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:12:12.796993 2136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7d83789b294c80b971f67fe4151dd8, State: Running, Role: LEADER
I20260812 01:12:12.797739 2136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7d83789b294c80b971f67fe4151dd8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34279 } }
I20260812 01:12:12.801059 2141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e7d83789b294c80b971f67fe4151dd8. Latest consensus state: current_term: 1 leader_uuid: "7e7d83789b294c80b971f67fe4151dd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7d83789b294c80b971f67fe4151dd8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34279 } } }
I20260812 01:12:12.801517 2141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:12.802134 2139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e7d83789b294c80b971f67fe4151dd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7d83789b294c80b971f67fe4151dd8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34279 } } }
I20260812 01:12:12.802441 2139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:12.802485 2168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:12:12.810338 2168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:12:12.822634 2168 catalog_manager.cc:1347] Generated new cluster ID: 88b02e41239a46eab57e6bdde5518c31
I20260812 01:12:12.822767 2168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:12:12.857604 2168 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:12:12.858963 2168 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:12:12.865554 2168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8: Generated new TSK 0
I20260812 01:12:12.866451 2168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab.
Entry for principal oryx/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab.
I20260812 01:12:12.950543 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34279
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab
--principal=oryx/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:13.301360 2181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:13.301666 2181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:13.301800 2181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:13.309535 2181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:13.309669 2181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:13.309799 2181 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 01:12:13.309909 2181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:12:13.320091 2181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab
--principal=oryx/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.151.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:13.321391 2181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:13.322795 2181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:13.335722 2189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:13.335800 2191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:13.335764 2195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:13.336850 2181 server_base.cc:1034] running on GCE node
I20260812 01:12:13.337474 2181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:13.338402 2181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:13.339629 2181 hybrid_clock.cc:648] HybridClock initialized: now 1786497133339546 us; error 52 us; skew 500 ppm
I20260812 01:12:13.339938 2181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:13 dist-test-slave-21b9 krb5kdc[1949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497133, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:13.343020 2181 init.cc:377] Logged in from keytab as oryx/127.20.151.1@KRBTEST.COM (short username oryx)
I20260812 01:12:13.344436 2181 webserver.cc:460] Webserver started at http://127.20.151.1:36775/ using document root <none> and password file <none>
I20260812 01:12:13.345088 2181 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:13.345176 2181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:13.345450 2181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:13.348337 2181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "82bb9adfa45144f4921bd16595f458ab"
format_stamp: "Formatted at 2026-08-12 01:12:13 on dist-test-slave-21b9"
server_key: "8d5119d70e0257aba52dfeb3a664bf79"
server_key_iv: "d94568b2ec14fe4cebec10a00404a9ae"
server_key_version: "encryptionkey@0"
I20260812 01:12:13.349120 2181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "82bb9adfa45144f4921bd16595f458ab"
format_stamp: "Formatted at 2026-08-12 01:12:13 on dist-test-slave-21b9"
server_key: "8d5119d70e0257aba52dfeb3a664bf79"
server_key_iv: "d94568b2ec14fe4cebec10a00404a9ae"
server_key_version: "encryptionkey@0"
I20260812 01:12:13.354166 2181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:12:13.357520 2205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:13.358779 2181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:12:13.358922 2181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "82bb9adfa45144f4921bd16595f458ab"
format_stamp: "Formatted at 2026-08-12 01:12:13 on dist-test-slave-21b9"
server_key: "8d5119d70e0257aba52dfeb3a664bf79"
server_key_iv: "d94568b2ec14fe4cebec10a00404a9ae"
server_key_version: "encryptionkey@0"
I20260812 01:12:13.359122 2181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:13.374392 2181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:13.378571 2181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:13.378829 2181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:13.381521 2181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:13.381599 2181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:13.381762 2181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:13.381819 2181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 01:12:13 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497133, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:13.404806 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:13.382174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:60747 (local address 127.20.151.62:34279)
0812 01:12:13.382727 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:13.382734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:13.383947 (+ 1213us) server_negotiation.cc:407] Connection header received
0812 01:12:13.385968 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:13.385974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:13.386063 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:13.386158 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:13.388597 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.389610 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.392063 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.392544 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.392697 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:13.397543 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:13.397551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:13.397556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:13.397602 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:13.400901 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:13.401869 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:13.401875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:13.401878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:13.402045 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:13.402704 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:13.402709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:13.402712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:13.403056 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:13.403200 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:13.404210 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0812 01:12:13.404441 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":312,"threads_started":1}
I20260812 01:12:13.406267 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:13.382823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34279 (local address 127.20.151.1:60747)
0812 01:12:13.383777 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:13.383822 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:13.385699 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:13.386432 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:13.386444 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:13.387431 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:13.388377 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:13.388417 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.389792 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.389800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:13.391908 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:13.391924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.392855 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.392863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:13.393196 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:13.394046 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:13.394074 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:13.397266 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:13.401053 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:13.401066 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:13.401080 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:13.401719 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:13.402202 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:13.402208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:13.402211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:13.402533 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:13.403200 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:13.403208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:13.403712 (+ 504us) client_negotiation.cc:769] Sending connection context
0812 01:12:13.404091 (+ 379us) client_negotiation.cc:240] Negotiation successful
0812 01:12:13.404702 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":365,"threads_started":1}
I20260812 01:12:13.426187 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:13.412790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34279 (local address 127.20.151.1:56315)
0812 01:12:13.413456 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:13.413476 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:13.413622 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:13.413909 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:13.413914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:13.414501 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:13.414909 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:13.414922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.416128 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.416132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:13.417373 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:13.417390 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.418262 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.418268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:13.418424 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:13.419089 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:13.419106 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:13.420132 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:13.423557 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:13.423561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:13.423564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:13.424156 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:13.424527 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:13.424532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:13.424536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:13.424661 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:13.425175 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:13.425179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:13.425400 (+ 221us) client_negotiation.cc:769] Sending connection context
0812 01:12:13.425573 (+ 173us) client_negotiation.cc:240] Negotiation successful
0812 01:12:13.425847 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":340,"threads_started":1}
I20260812 01:12:13.426400 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:13.412795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:56315 (local address 127.20.151.62:34279)
0812 01:12:13.413003 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:13.413009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:13.413561 (+ 552us) server_negotiation.cc:407] Connection header received
0812 01:12:13.413736 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:13.413740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:13.413801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:13.413869 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:13.415056 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.415982 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.417524 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.418070 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.418185 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:13.420383 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:13.420390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:13.420394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:13.420434 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:13.423395 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:13.424270 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:13.424276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:13.424280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:13.424418 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:13.424765 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:13.424770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:13.424774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:13.425049 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:13.425185 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:13.425739 (+ 554us) server_negotiation.cc:299] Negotiation successful
0812 01:12:13.426019 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 01:12:13.427175 2181 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:38229
I20260812 01:12:13.427220 2413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:38229 every 8 connection(s)
I20260812 01:12:13.428505 2181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:12:13.430622 2416 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34279
I20260812 01:12:13.430837 2416 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:13.431041 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 2181
I20260812 01:12:13.431252 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:12:13.431546 2416 heartbeater.cc:511] Master 127.20.151.62:34279 requested a full tablet report, sending...
I20260812 01:12:13.431955 21084 external_mini_cluster.cc:1428] Setting key a77b33fd24287d818f07d4998c4e9553
I20260812 01:12:13.433461 2043 ts_manager.cc:194] Registered new tserver with Master: 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229)
I20260812 01:12:13.434970 2043 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:60747
WARNING: no policy specified for oryx/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab.
Entry for principal oryx/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab.
I20260812 01:12:13.492816 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34279
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab
--principal=oryx/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:13.847338 2424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:13.847688 2424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:13.847858 2424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:13.855064 2424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:13.855172 2424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:13.855320 2424 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 01:12:13.855432 2424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:12:13.865432 2424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab
--principal=oryx/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.151.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:13.866768 2424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:13.868114 2424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:13.880734 2434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:13.880820 2437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:13.880941 2439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:13.881568 2424 server_base.cc:1034] running on GCE node
I20260812 01:12:13.882155 2424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:13.883033 2424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:13.884282 2424 hybrid_clock.cc:648] HybridClock initialized: now 1786497133884177 us; error 72 us; skew 500 ppm
I20260812 01:12:13.884615 2424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:13 dist-test-slave-21b9 krb5kdc[1949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497133, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:13.887748 2424 init.cc:377] Logged in from keytab as oryx/127.20.151.2@KRBTEST.COM (short username oryx)
I20260812 01:12:13.889127 2424 webserver.cc:460] Webserver started at http://127.20.151.2:38127/ using document root <none> and password file <none>
I20260812 01:12:13.889832 2424 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:13.889921 2424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:13.890206 2424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:13.893080 2424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "64d53f0dc4c74fbea95d45ea5474c78e"
format_stamp: "Formatted at 2026-08-12 01:12:13 on dist-test-slave-21b9"
server_key: "c08ee680a8e6c109a0be0ec2e84c6752"
server_key_iv: "de69f9a6e0dda33acc3ad70756934715"
server_key_version: "encryptionkey@0"
I20260812 01:12:13.893832 2424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "64d53f0dc4c74fbea95d45ea5474c78e"
format_stamp: "Formatted at 2026-08-12 01:12:13 on dist-test-slave-21b9"
server_key: "c08ee680a8e6c109a0be0ec2e84c6752"
server_key_iv: "de69f9a6e0dda33acc3ad70756934715"
server_key_version: "encryptionkey@0"
I20260812 01:12:13.898939 2424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 01:12:13.902715 2447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:13.904151 2424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 01:12:13.904294 2424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "64d53f0dc4c74fbea95d45ea5474c78e"
format_stamp: "Formatted at 2026-08-12 01:12:13 on dist-test-slave-21b9"
server_key: "c08ee680a8e6c109a0be0ec2e84c6752"
server_key_iv: "de69f9a6e0dda33acc3ad70756934715"
server_key_version: "encryptionkey@0"
I20260812 01:12:13.904490 2424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:13.930625 2424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:13.935101 2424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:13.935415 2424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:13.938205 2424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:13.938300 2424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:13.938462 2424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:13.938530 2424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 01:12:13 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497133, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:13.961992 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:13.938945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:52201 (local address 127.20.151.62:34279)
0812 01:12:13.939476 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:13.939482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:13.940737 (+ 1255us) server_negotiation.cc:407] Connection header received
0812 01:12:13.942938 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:13.942944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:13.943051 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:13.943131 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:13.945719 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.946778 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.949095 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.949710 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.949831 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:13.954716 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:13.954722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:13.954725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:13.954760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:13.958199 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:13.959217 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:13.959223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:13.959227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:13.959430 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:13.960053 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:13.960057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:13.960060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:13.960404 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:13.960548 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:13.961335 (+ 787us) server_negotiation.cc:299] Negotiation successful
0812 01:12:13.961588 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":253,"threads_started":1}
I20260812 01:12:13.963102 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:13.939645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34279 (local address 127.20.151.2:52201)
0812 01:12:13.940552 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:13.940609 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:13.942560 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:13.943765 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:13.943777 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:13.944610 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:13.945512 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:13.945544 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.946942 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.946946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:13.948928 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:13.948952 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.949981 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.949989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:13.950254 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:13.951042 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:13.951080 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:13.954482 (+ 3402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:13.958364 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:13.958375 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:13.958397 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:13.959084 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:13.959605 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:13.959611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:13.959615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:13.959931 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:13.960554 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:13.960563 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:13.960936 (+ 373us) client_negotiation.cc:769] Sending connection context
0812 01:12:13.961192 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 01:12:13.961748 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":373,"threads_started":1}
I20260812 01:12:13.984663 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:13.969740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:47931 (local address 127.20.151.62:34279)
0812 01:12:13.969940 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:13.969947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:13.970548 (+ 601us) server_negotiation.cc:407] Connection header received
0812 01:12:13.970742 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:13.970746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:13.970824 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:13.970906 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:13.972422 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.973313 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.975615 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.976131 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.976267 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:13.978605 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:13.978610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:13.978613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:13.978645 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:13.981490 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:13.982452 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:13.982457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:13.982461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:13.982628 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:13.983095 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:13.983100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:13.983104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:13.983533 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:13.983742 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:13.984230 (+ 488us) server_negotiation.cc:299] Negotiation successful
0812 01:12:13.984412 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 01:12:13.985107 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:13.969740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34279 (local address 127.20.151.2:47931)
0812 01:12:13.970419 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:13.970443 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:13.970628 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:13.970952 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:13.970957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:13.971714 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:13.972224 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:13.972243 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.973454 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.973461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:13.975446 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:13.975465 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:13.976351 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:13.976358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:13.976540 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:13.977195 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:13.977219 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:13.978394 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:13.981631 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:13.981641 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:13.981645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:13.982317 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:13.982781 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:13.982787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:13.982791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:13.982934 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:13.983660 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:13.983666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:13.983943 (+ 277us) client_negotiation.cc:769] Sending connection context
0812 01:12:13.984143 (+ 200us) client_negotiation.cc:240] Negotiation successful
0812 01:12:13.984528 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":342,"threads_started":1}
I20260812 01:12:13.990069 2424 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:34849
I20260812 01:12:13.990091 2664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:34849 every 8 connection(s)
I20260812 01:12:13.991250 2424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:12:13.993214 2667 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34279
I20260812 01:12:13.993431 2667 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:13.994043 2667 heartbeater.cc:511] Master 127.20.151.62:34279 requested a full tablet report, sending...
I20260812 01:12:13.995386 2043 ts_manager.cc:194] Registered new tserver with Master: 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849)
I20260812 01:12:13.996177 2043 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:52201
I20260812 01:12:14.001372 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 2424
I20260812 01:12:14.001549 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:12:14.001937 21084 external_mini_cluster.cc:1428] Setting key eaa4ccaa82cceb238a9424e8c2664d78
WARNING: no policy specified for oryx/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab.
Entry for principal oryx/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab.
I20260812 01:12:14.066876 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34279
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab
--principal=oryx/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:14.409533 2675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:14.409857 2675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:14.410051 2675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:14.420869 2675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:14.421002 2675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:14.421144 2675 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 01:12:14.421265 2675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:12:14.435886 2675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab
--principal=oryx/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.151.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:14.437613 2675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:14.438042 2416 heartbeater.cc:503] Master 127.20.151.62:34279 was elected leader, sending a full tablet report...
I20260812 01:12:14.439632 2675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:14.455420 2684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:14.455648 2681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:14.455816 2682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:14.456295 2675 server_base.cc:1034] running on GCE node
I20260812 01:12:14.456933 2675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:14.457906 2675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:14.459139 2675 hybrid_clock.cc:648] HybridClock initialized: now 1786497134459051 us; error 62 us; skew 500 ppm
I20260812 01:12:14.459582 2675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:14 dist-test-slave-21b9 krb5kdc[1949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497134, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:14.463982 2675 init.cc:377] Logged in from keytab as oryx/127.20.151.3@KRBTEST.COM (short username oryx)
I20260812 01:12:14.465519 2675 webserver.cc:460] Webserver started at http://127.20.151.3:40407/ using document root <none> and password file <none>
I20260812 01:12:14.466300 2675 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:14.466396 2675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:14.466799 2675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:14.470045 2675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "9fa7dff587e8483686bcc45c6102dd8b"
format_stamp: "Formatted at 2026-08-12 01:12:14 on dist-test-slave-21b9"
server_key: "14a19b6407172ff1b76818cb70777632"
server_key_iv: "3181f65016e8eee9095a843721c9c5d8"
server_key_version: "encryptionkey@0"
I20260812 01:12:14.470830 2675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "9fa7dff587e8483686bcc45c6102dd8b"
format_stamp: "Formatted at 2026-08-12 01:12:14 on dist-test-slave-21b9"
server_key: "14a19b6407172ff1b76818cb70777632"
server_key_iv: "3181f65016e8eee9095a843721c9c5d8"
server_key_version: "encryptionkey@0"
I20260812 01:12:14.476159 2675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 01:12:14.479782 2696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:14.481132 2675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 01:12:14.481297 2675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "9fa7dff587e8483686bcc45c6102dd8b"
format_stamp: "Formatted at 2026-08-12 01:12:14 on dist-test-slave-21b9"
server_key: "14a19b6407172ff1b76818cb70777632"
server_key_iv: "3181f65016e8eee9095a843721c9c5d8"
server_key_version: "encryptionkey@0"
I20260812 01:12:14.481509 2675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:14.494671 2675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:14.499341 2675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:14.499608 2675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:14.502746 2675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:14.502838 2675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:14.502985 2675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:14.503064 2675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 01:12:14 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497134, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:14.527878 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:14.503466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:39611 (local address 127.20.151.62:34279)
0812 01:12:14.504008 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:14.504014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:14.505171 (+ 1157us) server_negotiation.cc:407] Connection header received
0812 01:12:14.508174 (+ 3003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:14.508180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:14.508285 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:14.508417 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:14.511070 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:14.512217 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:14.514614 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:14.515192 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:14.515336 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:14.520367 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:14.520374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:14.520380 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:14.520424 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:14.524054 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:14.525036 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:14.525043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:14.525047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:14.525245 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:14.525814 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:14.525819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:14.525822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:14.526205 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:14.526353 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:14.527122 (+ 769us) server_negotiation.cc:299] Negotiation successful
0812 01:12:14.527421 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":266,"threads_started":1}
I20260812 01:12:14.528961 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:14.504093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34279 (local address 127.20.151.3:39611)
0812 01:12:14.504957 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:14.505026 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:14.507729 (+ 2703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:14.508725 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:14.508738 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:14.509781 (+ 1043us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:14.510845 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:14.510875 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:14.512383 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:14.512389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:14.514457 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:14.514472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:14.515516 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:14.515525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:14.515830 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:14.516655 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:14.516689 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:14.520088 (+ 3399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:14.524252 (+ 4164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:14.524263 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:14.524287 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:14.524898 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:14.525412 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:14.525417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:14.525419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:14.525665 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:14.526346 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:14.526352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:14.526770 (+ 418us) client_negotiation.cc:769] Sending connection context
0812 01:12:14.527022 (+ 252us) client_negotiation.cc:240] Negotiation successful
0812 01:12:14.527606 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":338,"threads_started":1}
I20260812 01:12:14.551002 2675 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:41959
I20260812 01:12:14.551034 2919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:41959 every 8 connection(s)
I20260812 01:12:14.552675 2675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:12:14.554925 2920 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34279
I20260812 01:12:14.555120 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:14.539934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34279 (local address 127.20.151.3:33447)
0812 01:12:14.540589 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:14.540613 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:14.540792 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:14.541232 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:14.541238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:14.541783 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:14.542111 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:14.542126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:14.543639 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:14.543645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:14.545027 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:14.545039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:14.546023 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:14.546031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:14.546224 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:14.546942 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:14.546970 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:14.548233 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:14.552258 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:14.552262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:14.552265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:14.552712 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:14.553090 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:14.553095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:14.553098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:14.553264 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:14.553979 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:14.553984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:14.554220 (+ 236us) client_negotiation.cc:769] Sending connection context
0812 01:12:14.554361 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 01:12:14.554713 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":351,"threads_started":1}
I20260812 01:12:14.555212 2920 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:14.555887 2920 heartbeater.cc:511] Master 127.20.151.62:34279 requested a full tablet report, sending...
I20260812 01:12:14.555949 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:14.539592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:33447 (local address 127.20.151.62:34279)
0812 01:12:14.539815 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:14.539822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:14.540710 (+ 888us) server_negotiation.cc:407] Connection header received
0812 01:12:14.540938 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:14.540944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:14.541038 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:14.541156 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:14.542286 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:14.543448 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:14.545175 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:14.545830 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:14.545967 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:14.548477 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:14.548484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:14.548489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:14.548548 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:14.552127 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:14.552834 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:14.552840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:14.552844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:14.552981 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:14.553426 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:14.553431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:14.553433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:14.553841 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:14.555420 (+ 1579us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:14.555534 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 01:12:14.555716 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:12:14.556670 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 2675
I20260812 01:12:14.556892 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:12:14.557390 2040 ts_manager.cc:194] Registered new tserver with Master: 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3:41959)
I20260812 01:12:14.557495 21084 external_mini_cluster.cc:1428] Setting key 3e8bb14e2d3d05db9d4232e15a5d5c18
I20260812 01:12:14.558305 2040 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:39611
I20260812 01:12:14.562477 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 01:12:14 dist-test-slave-21b9 krb5kdc[1949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 01:12:14 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:14 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:14.998509 2667 heartbeater.cc:503] Master 127.20.151.62:34279 was elected leader, sending a full tablet report...
I20260812 01:12:15.560870 2920 heartbeater.cc:503] Master 127.20.151.62:34279 was elected leader, sending a full tablet report...
W20260812 01:12:15.652026 2935 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.031s user 0.000s sys 0.020s
W20260812 01:12:15.653136 2711 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:14.615339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.20.151.62:34279)
0812 01:12:14.615576 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:14.615583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:14.615802 (+ 219us) server_negotiation.cc:407] Connection header received
0812 01:12:14.615970 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:14.615974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:14.616069 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:14.616157 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:14.617526 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:14.618263 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:14.619654 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:14.620078 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:14.620183 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.652903 (+1032720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58038: BlockingRecv error: recv got EOF from 127.0.0.1:58038 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 12 01:12:15 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:15.674011 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.660676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.20.151.62:34279)
0812 01:12:15.660903 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:15.660909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:15.661068 (+ 159us) server_negotiation.cc:407] Connection header received
0812 01:12:15.661247 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:15.661251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:15.661318 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:15.661400 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:15.662659 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.663454 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.664706 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.665163 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.665269 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.669131 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:15.669136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:15.669140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:15.669189 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:15.671795 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:15.672397 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:15.672400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:15.672403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:15.672518 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:15.672791 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:15.672794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:15.672796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:15.673053 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:15.673248 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:15.673496 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 01:12:15.673700 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 01:12:15.678102 2040 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:12:15.685173 2040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:12:15.720477 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.698597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.20.151.2:34849)
0812 01:12:15.712065 (+ 13468us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:15.712075 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:15.712112 (+ 37us) server_negotiation.cc:407] Connection header received
0812 01:12:15.712232 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:15.712242 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:15.712539 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:15.712769 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:15.713967 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.715615 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.717867 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.719396 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.719539 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.719737 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:15.719981 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 01:12:15.720174 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13268,"thread_start_us":15073,"threads_started":1}
I20260812 01:12:15.720547 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.698003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:41959 (local address 127.0.0.1:49322)
0812 01:12:15.708044 (+ 10041us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:15.708104 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:15.708333 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:15.712803 (+ 4470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:15.712812 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:15.712841 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:15.713277 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:15.713289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.714912 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.714915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:15.717172 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:15.717188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.719319 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.719326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:15.719594 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.719657 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 01:12:15.719800 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 01:12:15.720037 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9696,"mutex_wait_us":28,"thread_start_us":12016,"threads_started":1}
I20260812 01:12:15.721446 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.698281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.20.151.3:41959)
0812 01:12:15.712304 (+ 14023us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:15.712310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:15.712332 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:12:15.712408 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:15.712413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:15.712579 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:15.712760 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:15.713444 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.714730 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.717335 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.718792 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.718922 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.719151 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:15.720980 (+ 1829us) server_negotiation.cc:299] Negotiation successful
0812 01:12:15.721151 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13885,"thread_start_us":14645,"threads_started":1}
I20260812 01:12:15.720549 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.698187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:34849 (local address 127.0.0.1:48682)
0812 01:12:15.710520 (+ 12333us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:15.710544 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:15.710663 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:15.713020 (+ 2357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:15.713026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:15.713058 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:15.713542 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:15.713554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.715741 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.715746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:15.717690 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:15.717704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.719618 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.719624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:15.719805 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.719833 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 01:12:15.719923 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 01:12:15.720058 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12195,"mutex_wait_us":24,"thread_start_us":11755,"threads_started":1}
I20260812 01:12:15.723522 2959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.700654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:38229 (local address 127.0.0.1:38660)
0812 01:12:15.710550 (+ 9896us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:15.710565 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:15.710673 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:15.716435 (+ 5762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:15.716440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:15.716468 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:15.716975 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:15.716989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.718827 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.718833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:15.720955 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:15.720967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.722681 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.722687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:15.722854 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.722886 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:12:15.722968 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 01:12:15.723095 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9796,"mutex_wait_us":24,"spinlock_wait_cycles":18688,"thread_start_us":6614,"threads_started":1}
I20260812 01:12:15.724429 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.701291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.20.151.1:38229)
0812 01:12:15.715787 (+ 14496us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:15.715796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:15.715827 (+ 31us) server_negotiation.cc:407] Connection header received
0812 01:12:15.715928 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:15.715936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:15.716212 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:15.716415 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:15.717170 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.718663 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.721123 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.722464 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.722592 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.722792 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:15.723954 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0812 01:12:15.724224 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14316,"thread_start_us":14147,"threads_started":1}
I20260812 01:12:15.725520 2806 tablet_service.cc:1467] Processing CreateTablet for tablet f60f6581f1364434b173a8ece11b9111 (DEFAULT_TABLE table=test-table [id=11e3eb9915c644eda51f75805549eaa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:15.725520 2543 tablet_service.cc:1467] Processing CreateTablet for tablet f60f6581f1364434b173a8ece11b9111 (DEFAULT_TABLE table=test-table [id=11e3eb9915c644eda51f75805549eaa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:15.727042 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60f6581f1364434b173a8ece11b9111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:15.727053 2806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60f6581f1364434b173a8ece11b9111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:15.727196 2297 tablet_service.cc:1467] Processing CreateTablet for tablet f60f6581f1364434b173a8ece11b9111 (DEFAULT_TABLE table=test-table [id=11e3eb9915c644eda51f75805549eaa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:15.728477 2297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60f6581f1364434b173a8ece11b9111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:15.740976 2970 tablet_bootstrap.cc:492] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: Bootstrap starting.
I20260812 01:12:15.744047 2973 tablet_bootstrap.cc:492] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: Bootstrap starting.
I20260812 01:12:15.744746 2970 tablet_bootstrap.cc:654] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:15.745749 2971 tablet_bootstrap.cc:492] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e: Bootstrap starting.
I20260812 01:12:15.746346 2970 log.cc:824] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:15.749094 2970 tablet_bootstrap.cc:492] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: No bootstrap required, opened a new log
I20260812 01:12:15.749366 2970 ts_tablet_manager.cc:1397] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 01:12:15.749406 2971 tablet_bootstrap.cc:654] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:15.749522 2973 tablet_bootstrap.cc:654] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:15.750684 2973 log.cc:824] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:15.750681 2971 log.cc:824] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:15.753615 2971 tablet_bootstrap.cc:492] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e: No bootstrap required, opened a new log
I20260812 01:12:15.753892 2971 ts_tablet_manager.cc:1397] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260812 01:12:15.757647 2970 raft_consensus.cc:348] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } }
I20260812 01:12:15.757994 2970 raft_consensus.cc:374] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:15.758085 2970 raft_consensus.cc:729] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82bb9adfa45144f4921bd16595f458ab, State: Initialized, Role: FOLLOWER
I20260812 01:12:15.758671 2970 consensus_queue.cc:260] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } }
I20260812 01:12:15.760376 2970 ts_tablet_manager.cc:1428] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260812 01:12:15.763706 2973 tablet_bootstrap.cc:492] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: No bootstrap required, opened a new log
I20260812 01:12:15.764012 2973 ts_tablet_manager.cc:1397] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.008s
I20260812 01:12:15.764321 2971 raft_consensus.cc:348] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } }
I20260812 01:12:15.764777 2971 raft_consensus.cc:374] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:15.764884 2971 raft_consensus.cc:729] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d53f0dc4c74fbea95d45ea5474c78e, State: Initialized, Role: FOLLOWER
I20260812 01:12:15.765537 2971 consensus_queue.cc:260] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } }
I20260812 01:12:15.767134 2971 ts_tablet_manager.cc:1428] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260812 01:12:15.774353 2973 raft_consensus.cc:348] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } }
I20260812 01:12:15.774735 2973 raft_consensus.cc:374] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:15.774811 2973 raft_consensus.cc:729] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa7dff587e8483686bcc45c6102dd8b, State: Initialized, Role: FOLLOWER
I20260812 01:12:15.775346 2973 consensus_queue.cc:260] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } }
I20260812 01:12:15.776734 2973 ts_tablet_manager.cc:1428] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: Time spent starting tablet: real 0.013s user 0.009s sys 0.001s
W20260812 01:12:15.805823 2922 tablet.cc:2369] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:12:15.911238 2982 raft_consensus.cc:482] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:12:15.911607 2982 raft_consensus.cc:504] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } }
I20260812 01:12:15.913372 2982 leader_election.cc:288] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849), 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229)
I20260812 01:12:15.920702 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.913939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:36563 (local address 127.20.151.1:38229)
0812 01:12:15.914217 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:15.914228 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:15.914736 (+ 508us) server_negotiation.cc:407] Connection header received
0812 01:12:15.914939 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:15.914946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:15.915024 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:15.915107 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:15.915828 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.916524 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.918909 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.919772 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.919938 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.920010 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:15.920343 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 01:12:15.920472 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 01:12:15.920701 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.913716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:57009 (local address 127.20.151.2:34849)
0812 01:12:15.913928 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:15.913934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:15.914809 (+ 875us) server_negotiation.cc:407] Connection header received
0812 01:12:15.914959 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:15.914963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:15.915035 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:15.915118 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:15.915818 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.916529 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.918940 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.919733 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.919861 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.919938 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:15.920300 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 01:12:15.920441 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 01:12:15.920888 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.913665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:38229 (local address 127.20.151.3:36563)
0812 01:12:15.914616 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:15.914633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:15.914771 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:15.915146 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:15.915150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:15.915249 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:15.915678 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:15.915692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.916659 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.916664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:15.918738 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:15.918754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.920014 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.920019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:15.920188 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.920211 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 01:12:15.920300 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 01:12:15.920447 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":172,"spinlock_wait_cycles":22912,"thread_start_us":483,"threads_started":1}
I20260812 01:12:15.921228 2987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.913765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:34849 (local address 127.20.151.3:57009)
0812 01:12:15.914675 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:15.914692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:15.914828 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:15.915147 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:15.915151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:15.915175 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:15.915645 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:15.915657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.916655 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.916660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:15.918764 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:15.918777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.919959 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.919966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:15.920137 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.920173 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 01:12:15.920248 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 01:12:15.920427 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":98,"spinlock_wait_cycles":36096,"thread_start_us":495,"threads_started":1}
I20260812 01:12:15.921883 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60f6581f1364434b173a8ece11b9111" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82bb9adfa45144f4921bd16595f458ab" is_pre_election: true
I20260812 01:12:15.922073 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60f6581f1364434b173a8ece11b9111" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" is_pre_election: true
I20260812 01:12:15.922406 2331 raft_consensus.cc:2454] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 0.
I20260812 01:12:15.922564 2583 raft_consensus.cc:2454] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 0.
I20260812 01:12:15.923089 2701 leader_election.cc:302] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 64d53f0dc4c74fbea95d45ea5474c78e, 82bb9adfa45144f4921bd16595f458ab, 9fa7dff587e8483686bcc45c6102dd8b; no voters:
I20260812 01:12:15.923586 2982 raft_consensus.cc:2781] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:12:15.923748 2982 raft_consensus.cc:482] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:12:15.923857 2982 raft_consensus.cc:3037] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:15.926043 2982 raft_consensus.cc:504] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } }
I20260812 01:12:15.926734 2982 leader_election.cc:288] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 election: Requested vote from peers 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849), 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229)
I20260812 01:12:15.927139 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60f6581f1364434b173a8ece11b9111" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d53f0dc4c74fbea95d45ea5474c78e"
I20260812 01:12:15.927393 2583 raft_consensus.cc:3037] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:15.927461 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60f6581f1364434b173a8ece11b9111" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82bb9adfa45144f4921bd16595f458ab"
I20260812 01:12:15.927686 2331 raft_consensus.cc:3037] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:15.929689 2583 raft_consensus.cc:2454] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 1.
I20260812 01:12:15.929909 2331 raft_consensus.cc:2454] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 1.
I20260812 01:12:15.930110 2700 leader_election.cc:302] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d53f0dc4c74fbea95d45ea5474c78e, 9fa7dff587e8483686bcc45c6102dd8b; no voters:
I20260812 01:12:15.930438 2982 raft_consensus.cc:2781] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:15.930918 2982 raft_consensus.cc:686] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 1 LEADER]: Becoming Leader. State: Replica: 9fa7dff587e8483686bcc45c6102dd8b, State: Running, Role: LEADER
I20260812 01:12:15.931473 2982 consensus_queue.cc:237] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } }
W20260812 01:12:15.932036 2417 tablet.cc:2369] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:12:15.935978 2040 catalog_manager.cc:5495] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3). New cstate: current_term: 1 leader_uuid: "9fa7dff587e8483686bcc45c6102dd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } health_report { overall_health: UNKNOWN } } }
I20260812 01:12:15.955372 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:15.950157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.20.151.3:41959)
0812 01:12:15.950364 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:15.950370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:15.950391 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:12:15.950529 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:15.950533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:15.950634 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:15.950764 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:12:15.951125 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.951896 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.953228 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:15.953646 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:15.953755 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:15.954229 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:12:15.954834 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:12:15.954937 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:15.955012 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 01:12:15.955081 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 01:12:15.963577 2331 raft_consensus.cc:1261] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Refusing update from remote peer 9fa7dff587e8483686bcc45c6102dd8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:15.963696 2583 raft_consensus.cc:1261] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Refusing update from remote peer 9fa7dff587e8483686bcc45c6102dd8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:15.964593 2988 consensus_queue.cc:1035] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:15.964946 2982 consensus_queue.cc:1035] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:15.981243 2993 mvcc.cc:205] Tried to move back new op lower bound from 7317492268896628736 to 7317492268779278336. Current Snapshot: MvccSnapshot[applied={T|T < 7317492268896628736}]
I20260812 01:12:15.986624 2995 mvcc.cc:205] Tried to move back new op lower bound from 7317492268896628736 to 7317492268779278336. Current Snapshot: MvccSnapshot[applied={T|T < 7317492268896628736}]
I20260812 01:12:15.998312 2040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:15.998627 2040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:16.003880 2040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8: Sending DeleteTablet for 3 replicas of tablet f60f6581f1364434b173a8ece11b9111
I20260812 01:12:16.005016 2806 tablet_service.cc:1514] Processing DeleteTablet for tablet f60f6581f1364434b173a8ece11b9111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:15 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:49322
I20260812 01:12:16.005209 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet f60f6581f1364434b173a8ece11b9111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:15 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:48682
I20260812 01:12:16.006956 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet f60f6581f1364434b173a8ece11b9111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:15 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:48682
I20260812 01:12:16.008366 2297 tablet_service.cc:1514] Processing DeleteTablet for tablet f60f6581f1364434b173a8ece11b9111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:15 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:38660
I20260812 01:12:16.016137 3018 tablet_replica.cc:330] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: stopping tablet replica
I20260812 01:12:16.016434 3013 tablet_replica.cc:330] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e: stopping tablet replica
I20260812 01:12:16.017390 3011 tablet_replica.cc:330] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: stopping tablet replica
I20260812 01:12:16.016736 3018 raft_consensus.cc:2229] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:16.017518 2040 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 01:12:16.018158 3013 raft_consensus.cc:2229] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 01:12:16.018285 2040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:12:16.018373 3011 raft_consensus.cc:2229] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:12:16.018711 3013 raft_consensus.cc:2258] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:16.019064 3011 raft_consensus.cc:2258] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:16.019716 3018 raft_consensus.cc:2258] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:16.021147 3013 ts_tablet_manager.cc:1905] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:16.021723 3018 ts_tablet_manager.cc:1905] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:16.027552 2805 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1615e861534e6297e061f41e84c68c (DEFAULT_TABLE table=test-table [id=7e3198dcea954460bd926dc179bb40d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:16.028216 2805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1615e861534e6297e061f41e84c68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:16.028398 3018 ts_tablet_manager.cc:1918] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:16.028575 3018 log.cc:1196] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/f60f6581f1364434b173a8ece11b9111
I20260812 01:12:16.028889 3013 ts_tablet_manager.cc:1918] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:16.029122 3018 ts_tablet_manager.cc:1939] T f60f6581f1364434b173a8ece11b9111 P 82bb9adfa45144f4921bd16595f458ab: Deleting consensus metadata
I20260812 01:12:16.029174 3013 log.cc:1196] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/f60f6581f1364434b173a8ece11b9111
I20260812 01:12:16.029814 3013 ts_tablet_manager.cc:1939] T f60f6581f1364434b173a8ece11b9111 P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting consensus metadata
I20260812 01:12:16.031250 2014 catalog_manager.cc:4841] TS 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229): tablet f60f6581f1364434b173a8ece11b9111 (table test-table [id=11e3eb9915c644eda51f75805549eaa3]) successfully deleted
I20260812 01:12:16.031821 2013 catalog_manager.cc:4841] TS 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849): tablet f60f6581f1364434b173a8ece11b9111 (table test-table [id=11e3eb9915c644eda51f75805549eaa3]) successfully deleted
I20260812 01:12:16.032953 3011 ts_tablet_manager.cc:1905] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 01:12:16.032975 2013 catalog_manager.cc:4808] TS 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849): delete failed for tablet f60f6581f1364434b173a8ece11b9111 because the tablet was not found. No further retry: Not found: Tablet not found: f60f6581f1364434b173a8ece11b9111
I20260812 01:12:16.035795 2297 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1615e861534e6297e061f41e84c68c (DEFAULT_TABLE table=test-table [id=7e3198dcea954460bd926dc179bb40d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:16.036398 2973 tablet_bootstrap.cc:492] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b: Bootstrap starting.
I20260812 01:12:16.036509 2297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1615e861534e6297e061f41e84c68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:16.038190 2973 tablet_bootstrap.cc:654] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:16.038691 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1615e861534e6297e061f41e84c68c (DEFAULT_TABLE table=test-table [id=7e3198dcea954460bd926dc179bb40d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:16.039533 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1615e861534e6297e061f41e84c68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:16.040169 3011 ts_tablet_manager.cc:1918] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:16.040376 3011 log.cc:1196] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/f60f6581f1364434b173a8ece11b9111
I20260812 01:12:16.040971 3011 ts_tablet_manager.cc:1939] T f60f6581f1364434b173a8ece11b9111 P 9fa7dff587e8483686bcc45c6102dd8b: Deleting consensus metadata
I20260812 01:12:16.043140 2018 catalog_manager.cc:4841] TS 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3:41959): tablet f60f6581f1364434b173a8ece11b9111 (table test-table [id=11e3eb9915c644eda51f75805549eaa3]) successfully deleted
I20260812 01:12:16.044250 2973 tablet_bootstrap.cc:492] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b: No bootstrap required, opened a new log
I20260812 01:12:16.044404 2973 ts_tablet_manager.cc:1397] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260812 01:12:16.045113 2973 raft_consensus.cc:348] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.045301 2973 raft_consensus.cc:374] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:16.045393 2973 raft_consensus.cc:729] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa7dff587e8483686bcc45c6102dd8b, State: Initialized, Role: FOLLOWER
I20260812 01:12:16.045667 2973 consensus_queue.cc:260] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.046375 2973 ts_tablet_manager.cc:1428] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 01:12:16.048187 2970 tablet_bootstrap.cc:492] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab: Bootstrap starting.
I20260812 01:12:16.049890 2970 tablet_bootstrap.cc:654] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:16.052836 2971 tablet_bootstrap.cc:492] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e: Bootstrap starting.
I20260812 01:12:16.053093 2970 tablet_bootstrap.cc:492] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab: No bootstrap required, opened a new log
I20260812 01:12:16.053252 2970 ts_tablet_manager.cc:1397] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 01:12:16.054096 2970 raft_consensus.cc:348] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.054332 2970 raft_consensus.cc:374] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:16.054426 2970 raft_consensus.cc:729] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82bb9adfa45144f4921bd16595f458ab, State: Initialized, Role: FOLLOWER
I20260812 01:12:16.054786 2971 tablet_bootstrap.cc:654] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:16.054747 2970 consensus_queue.cc:260] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.055313 2970 ts_tablet_manager.cc:1428] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 01:12:16.057325 2971 tablet_bootstrap.cc:492] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e: No bootstrap required, opened a new log
I20260812 01:12:16.057484 2971 ts_tablet_manager.cc:1397] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 01:12:16.058295 2971 raft_consensus.cc:348] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.058517 2971 raft_consensus.cc:374] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:16.058624 2971 raft_consensus.cc:729] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d53f0dc4c74fbea95d45ea5474c78e, State: Initialized, Role: FOLLOWER
I20260812 01:12:16.058887 2971 consensus_queue.cc:260] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.059554 2971 ts_tablet_manager.cc:1428] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 01:12:16.138468 2981 raft_consensus.cc:482] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:12:16.138698 2981 raft_consensus.cc:504] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.140337 2981 leader_election.cc:288] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229), 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3:41959)
I20260812 01:12:16.148064 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.141024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:43357 (local address 127.20.151.1:38229)
0812 01:12:16.141204 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:16.141210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:16.141562 (+ 352us) server_negotiation.cc:407] Connection header received
0812 01:12:16.141929 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:16.141933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:16.142007 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:16.142096 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:16.142612 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.143502 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.146117 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.147106 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.147310 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.147397 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:16.147658 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 01:12:16.147791 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 01:12:16.148137 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.140608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:38229 (local address 127.20.151.2:43357)
0812 01:12:16.141429 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:16.141451 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:16.141705 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:16.142128 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:16.142132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:16.142158 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:16.142459 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:16.142468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.143630 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.143636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:16.145931 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:16.145947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.147347 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.147352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:16.147504 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.147529 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:12:16.147621 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 01:12:16.147753 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":3,"thread_start_us":665,"threads_started":1}
I20260812 01:12:16.148767 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1615e861534e6297e061f41e84c68c" candidate_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82bb9adfa45144f4921bd16595f458ab" is_pre_election: true
I20260812 01:12:16.149026 2331 raft_consensus.cc:2454] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d53f0dc4c74fbea95d45ea5474c78e in term 0.
I20260812 01:12:16.149451 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.140773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:39553 (local address 127.20.151.3:41959)
0812 01:12:16.140960 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:16.140966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:16.142010 (+ 1044us) server_negotiation.cc:407] Connection header received
0812 01:12:16.142254 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:16.142260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:16.142337 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:16.142399 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:16.143198 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.144211 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.146600 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.147884 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.148048 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.148120 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:16.148959 (+ 839us) server_negotiation.cc:299] Negotiation successful
0812 01:12:16.149118 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:12:16.149803 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.140764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:41959 (local address 127.20.151.2:39553)
0812 01:12:16.141896 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:16.141915 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:16.142065 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:16.142597 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:16.142602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:16.142626 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:16.143008 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:16.143019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.144371 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.144375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:16.146431 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:16.146447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.148579 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.148586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:16.148775 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.148808 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 01:12:16.148904 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 01:12:16.149224 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":29,"thread_start_us":851,"threads_started":1}
I20260812 01:12:16.149734 2452 leader_election.cc:302] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d53f0dc4c74fbea95d45ea5474c78e, 82bb9adfa45144f4921bd16595f458ab; no voters:
I20260812 01:12:16.150135 2981 raft_consensus.cc:2781] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:12:16.150246 2981 raft_consensus.cc:482] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:12:16.150342 2981 raft_consensus.cc:3037] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:16.150766 2842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1615e861534e6297e061f41e84c68c" candidate_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa7dff587e8483686bcc45c6102dd8b" is_pre_election: true
I20260812 01:12:16.151232 2842 raft_consensus.cc:2454] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d53f0dc4c74fbea95d45ea5474c78e in term 0.
I20260812 01:12:16.153007 2981 raft_consensus.cc:504] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.153602 2981 leader_election.cc:288] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [CANDIDATE]: Term 1 election: Requested vote from peers 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229), 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3:41959)
I20260812 01:12:16.154086 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1615e861534e6297e061f41e84c68c" candidate_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82bb9adfa45144f4921bd16595f458ab"
I20260812 01:12:16.154088 2842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1615e861534e6297e061f41e84c68c" candidate_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa7dff587e8483686bcc45c6102dd8b"
I20260812 01:12:16.154307 2331 raft_consensus.cc:3037] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:16.154335 2842 raft_consensus.cc:3037] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:16.156819 2842 raft_consensus.cc:2454] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d53f0dc4c74fbea95d45ea5474c78e in term 1.
I20260812 01:12:16.157210 2460 leader_election.cc:302] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d53f0dc4c74fbea95d45ea5474c78e, 9fa7dff587e8483686bcc45c6102dd8b; no voters:
I20260812 01:12:16.157421 2331 raft_consensus.cc:2454] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d53f0dc4c74fbea95d45ea5474c78e in term 1.
I20260812 01:12:16.157471 2981 raft_consensus.cc:2781] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:16.158071 2981 raft_consensus.cc:686] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 LEADER]: Becoming Leader. State: Replica: 64d53f0dc4c74fbea95d45ea5474c78e, State: Running, Role: LEADER
I20260812 01:12:16.158509 2981 consensus_queue.cc:237] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.162200 2043 catalog_manager.cc:5495] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e reported cstate change: term changed from 0 to 1, leader changed from <none> to 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2). New cstate: current_term: 1 leader_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } health_report { overall_health: UNKNOWN } } }
Aug 12 01:12:16 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497132, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.62@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:16.212106 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.197793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34279 (local address 127.0.0.1:58064)
0812 01:12:16.198639 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:16.198656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:16.198801 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:16.199099 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:16.199103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:16.199959 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:16.200220 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:16.200231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.201352 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.201358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:16.202404 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:16.202417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.203133 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.203140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:16.203336 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.204006 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:16.204028 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:16.206504 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:16.209570 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:16.209583 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:16.209592 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:16.210025 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:16.210378 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:16.210384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:16.210388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:16.210528 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:16.211030 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:16.211038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:16.211211 (+ 173us) client_negotiation.cc:769] Sending connection context
0812 01:12:16.211385 (+ 174us) client_negotiation.cc:240] Negotiation successful
0812 01:12:16.211676 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":566,"threads_started":1}
I20260812 01:12:16.212190 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.197788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58064 (local address 127.20.151.62:34279)
0812 01:12:16.198499 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:16.198506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:16.198736 (+ 230us) server_negotiation.cc:407] Connection header received
0812 01:12:16.198911 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:16.198916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:16.198971 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:16.199059 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:16.200392 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.201227 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.202533 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.202962 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.203076 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.206713 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:16.206718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:16.206722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:16.206751 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:16.209445 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:16.210141 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:16.210147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:16.210150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:16.210277 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:16.210628 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:16.210633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:16.210637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:16.210901 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:16.211036 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:16.211491 (+ 455us) server_negotiation.cc:299] Negotiation successful
0812 01:12:16.211837 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":518,"thread_start_us":321,"threads_started":1}
I20260812 01:12:16.218247 2043 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:58064:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 01:12:16.219588 2043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:12:16.226804 2297 tablet_service.cc:1467] Processing CreateTablet for tablet 4600e5d5354948cb8e89e683a6a795d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e6a66c472bc45819b39807c4d52aa22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 01:12:16.227195 2805 tablet_service.cc:1467] Processing CreateTablet for tablet 4600e5d5354948cb8e89e683a6a795d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e6a66c472bc45819b39807c4d52aa22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 01:12:16.227471 2297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4600e5d5354948cb8e89e683a6a795d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:16.227747 2805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4600e5d5354948cb8e89e683a6a795d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:16.227799 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 4600e5d5354948cb8e89e683a6a795d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e6a66c472bc45819b39807c4d52aa22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 01:12:16.228374 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4600e5d5354948cb8e89e683a6a795d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:16.234678 2970 tablet_bootstrap.cc:492] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab: Bootstrap starting.
I20260812 01:12:16.236281 2970 tablet_bootstrap.cc:654] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:16.236662 2971 tablet_bootstrap.cc:492] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: Bootstrap starting.
I20260812 01:12:16.237350 2973 tablet_bootstrap.cc:492] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b: Bootstrap starting.
I20260812 01:12:16.238353 2971 tablet_bootstrap.cc:654] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:16.239133 2973 tablet_bootstrap.cc:654] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:16.240886 2971 tablet_bootstrap.cc:492] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: No bootstrap required, opened a new log
I20260812 01:12:16.241108 2971 ts_tablet_manager.cc:1397] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 01:12:16.241163 2970 tablet_bootstrap.cc:492] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab: No bootstrap required, opened a new log
I20260812 01:12:16.241320 2970 ts_tablet_manager.cc:1397] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20260812 01:12:16.241909 2970 raft_consensus.cc:348] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.241847 2971 raft_consensus.cc:348] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.242108 2971 raft_consensus.cc:374] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:16.242108 2970 raft_consensus.cc:374] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:16.242187 2970 raft_consensus.cc:729] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82bb9adfa45144f4921bd16595f458ab, State: Initialized, Role: FOLLOWER
I20260812 01:12:16.242187 2971 raft_consensus.cc:729] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d53f0dc4c74fbea95d45ea5474c78e, State: Initialized, Role: FOLLOWER
I20260812 01:12:16.242436 2970 consensus_queue.cc:260] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.242420 2971 consensus_queue.cc:260] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.242823 2981 tablet_replica.cc:439] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } }
I20260812 01:12:16.242853 2980 tablet_replica.cc:439] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } }
I20260812 01:12:16.243108 2981 tablet_replica.cc:442] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:12:16.243165 2980 tablet_replica.cc:442] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:12:16.243669 2970 ts_tablet_manager.cc:1428] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 01:12:16.245122 2973 tablet_bootstrap.cc:492] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b: No bootstrap required, opened a new log
I20260812 01:12:16.245287 2973 ts_tablet_manager.cc:1397] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260812 01:12:16.246510 2973 raft_consensus.cc:348] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.246778 2973 raft_consensus.cc:374] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:16.246868 2973 raft_consensus.cc:729] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa7dff587e8483686bcc45c6102dd8b, State: Initialized, Role: FOLLOWER
I20260812 01:12:16.247174 2973 consensus_queue.cc:260] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
W20260812 01:12:16.247808 2670 tablet.cc:2369] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:12:16.247644 3002 tablet_replica.cc:439] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } }
I20260812 01:12:16.248013 3002 tablet_replica.cc:442] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:12:16.248373 2971 ts_tablet_manager.cc:1428] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260812 01:12:16.248549 2973 ts_tablet_manager.cc:1428] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 01:12:16.500516 2981 raft_consensus.cc:482] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:12:16.500746 2981 raft_consensus.cc:504] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.501334 2981 leader_election.cc:288] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229), 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3:41959)
I20260812 01:12:16.501912 2842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4600e5d5354948cb8e89e683a6a795d7" candidate_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa7dff587e8483686bcc45c6102dd8b" is_pre_election: true
I20260812 01:12:16.501928 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4600e5d5354948cb8e89e683a6a795d7" candidate_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82bb9adfa45144f4921bd16595f458ab" is_pre_election: true
I20260812 01:12:16.502156 2842 raft_consensus.cc:2454] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d53f0dc4c74fbea95d45ea5474c78e in term 0.
I20260812 01:12:16.502200 2331 raft_consensus.cc:2454] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d53f0dc4c74fbea95d45ea5474c78e in term 0.
I20260812 01:12:16.502606 2460 leader_election.cc:302] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d53f0dc4c74fbea95d45ea5474c78e, 9fa7dff587e8483686bcc45c6102dd8b; no voters:
I20260812 01:12:16.502898 2981 raft_consensus.cc:2781] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:12:16.503013 2981 raft_consensus.cc:482] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:12:16.503123 2981 raft_consensus.cc:3037] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:16.505124 2981 raft_consensus.cc:504] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.505659 2981 leader_election.cc:288] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [CANDIDATE]: Term 1 election: Requested vote from peers 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229), 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3:41959)
I20260812 01:12:16.506059 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4600e5d5354948cb8e89e683a6a795d7" candidate_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82bb9adfa45144f4921bd16595f458ab"
I20260812 01:12:16.506162 2842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4600e5d5354948cb8e89e683a6a795d7" candidate_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa7dff587e8483686bcc45c6102dd8b"
I20260812 01:12:16.506273 2331 raft_consensus.cc:3037] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:16.506378 2842 raft_consensus.cc:3037] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:16.508364 2331 raft_consensus.cc:2454] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d53f0dc4c74fbea95d45ea5474c78e in term 1.
I20260812 01:12:16.508421 2842 raft_consensus.cc:2454] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d53f0dc4c74fbea95d45ea5474c78e in term 1.
I20260812 01:12:16.508834 2460 leader_election.cc:302] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d53f0dc4c74fbea95d45ea5474c78e, 9fa7dff587e8483686bcc45c6102dd8b; no voters:
I20260812 01:12:16.509115 2981 raft_consensus.cc:2781] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:16.509268 2981 raft_consensus.cc:686] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 LEADER]: Becoming Leader. State: Replica: 64d53f0dc4c74fbea95d45ea5474c78e, State: Running, Role: LEADER
I20260812 01:12:16.509506 2981 consensus_queue.cc:237] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.511009 3027 tablet_replica.cc:439] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: TxnStatusTablet state changed. Reason: New leader 64d53f0dc4c74fbea95d45ea5474c78e. Latest consensus state: current_term: 1 leader_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } }
I20260812 01:12:16.511345 3027 tablet_replica.cc:442] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: This TxnStatusTablet replica's current role is: LEADER
I20260812 01:12:16.511914 3061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 01:12:16.512120 3061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 01:12:16.512307 2040 catalog_manager.cc:5495] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e reported cstate change: term changed from 0 to 1, leader changed from <none> to 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2). New cstate: current_term: 1 leader_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } health_report { overall_health: UNKNOWN } } }
I20260812 01:12:16.550786 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.543953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:34849 (local address 127.0.0.1:48688)
0812 01:12:16.544183 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:16.544206 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:16.544352 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:16.545159 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:16.545162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:16.545172 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 01:12:16.545350 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:16.545357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.546848 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.546852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:16.547897 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:16.547910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.548953 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.548957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:16.549094 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.549169 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 01:12:16.550278 (+ 1109us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 01:12:16.550297 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 01:12:16.550362 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 01:12:16.550518 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 01:12:16.550967 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.544126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.20.151.2:34849)
0812 01:12:16.544710 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:16.544717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:16.544737 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:12:16.544850 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:16.544855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:16.544965 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:16.545156 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:12:16.545476 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.546697 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.548053 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.548799 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.548932 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.549378 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:12:16.550168 (+ 790us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:12:16.550334 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:16.550391 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 01:12:16.550491 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":290,"threads_started":1}
I20260812 01:12:16.556200 2331 raft_consensus.cc:1261] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Refusing update from remote peer 64d53f0dc4c74fbea95d45ea5474c78e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:16.556420 2842 raft_consensus.cc:1261] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Refusing update from remote peer 64d53f0dc4c74fbea95d45ea5474c78e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:16.557147 2981 consensus_queue.cc:1035] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [LEADER]: Connected to new peer: Peer: permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:16.557662 3027 consensus_queue.cc:1035] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:16.579219 2980 tablet_replica.cc:439] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab: TxnStatusTablet state changed. Reason: New leader 64d53f0dc4c74fbea95d45ea5474c78e. Latest consensus state: current_term: 1 leader_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } }
I20260812 01:12:16.579587 2980 tablet_replica.cc:442] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:12:16.580576 3002 tablet_replica.cc:439] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b: TxnStatusTablet state changed. Reason: New leader 64d53f0dc4c74fbea95d45ea5474c78e. Latest consensus state: current_term: 1 leader_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } }
I20260812 01:12:16.580863 3002 tablet_replica.cc:442] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:12:16.583773 3068 mvcc.cc:205] Tried to move back new op lower bound from 7317492271326380032 to 7317492271145009152. Current Snapshot: MvccSnapshot[applied={T|T < 7317492271326380032}]
I20260812 01:12:16.585633 3027 tablet_replica.cc:439] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } }
I20260812 01:12:16.585914 3027 tablet_replica.cc:442] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: This TxnStatusTablet replica's current role is: LEADER
I20260812 01:12:16.586134 3027 tablet_replica.cc:439] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } }
I20260812 01:12:16.586370 3027 tablet_replica.cc:442] T 4600e5d5354948cb8e89e683a6a795d7 P 64d53f0dc4c74fbea95d45ea5474c78e: This TxnStatusTablet replica's current role is: LEADER
I20260812 01:12:16.591780 2980 tablet_replica.cc:439] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } }
I20260812 01:12:16.592041 2980 tablet_replica.cc:442] T 4600e5d5354948cb8e89e683a6a795d7 P 82bb9adfa45144f4921bd16595f458ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:12:16.595814 3002 tablet_replica.cc:439] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } } }
I20260812 01:12:16.596074 3002 tablet_replica.cc:442] T 4600e5d5354948cb8e89e683a6a795d7 P 9fa7dff587e8483686bcc45c6102dd8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 01:12:16.601329 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.592564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.20.151.2:34849)
0812 01:12:16.592739 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:16.592746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:16.595233 (+ 2487us) server_negotiation.cc:407] Connection header received
0812 01:12:16.595520 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:16.595526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:16.595617 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:16.595714 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:12:16.596491 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.597644 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.599616 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.600067 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.600156 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.600658 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:12:16.600759 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:12:16.600823 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:16.600992 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 01:12:16.601056 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 01:12:16.617843 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.612150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:33961 (local address 127.20.151.2:34849)
0812 01:12:16.612320 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:16.612324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:16.613040 (+ 716us) server_negotiation.cc:407] Connection header received
0812 01:12:16.613227 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:16.613232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:16.613298 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:16.613372 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:12:16.613897 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.614665 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.616141 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.616561 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.616663 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.617083 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:12:16.617180 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:12:16.617258 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:16.617391 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 01:12:16.617469 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":46}
I20260812 01:12:16.617938 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.612237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:34849 (local address 127.20.151.2:33961)
0812 01:12:16.612938 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:16.612953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:16.613102 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:16.613394 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:16.613399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:16.613409 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 01:12:16.613743 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:16.613752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.614783 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.614788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:16.616015 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:16.616028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.616733 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.616738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:16.616861 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.616915 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 01:12:16.617270 (+ 355us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 01:12:16.617293 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 01:12:16.617360 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 01:12:16.617494 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":1,"thread_start_us":410,"threads_started":1}
I20260812 01:12:16.626986 2840 raft_consensus.cc:1261] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Refusing update from remote peer 64d53f0dc4c74fbea95d45ea5474c78e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:16.627352 2329 raft_consensus.cc:1261] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Refusing update from remote peer 64d53f0dc4c74fbea95d45ea5474c78e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:16.627553 3027 consensus_queue.cc:1035] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:16.628034 3027 consensus_queue.cc:1035] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [LEADER]: Connected to new peer: Peer: permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:16.721278 2043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:16.721536 2043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:16.724725 2043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8: Sending DeleteTablet for 3 replicas of tablet 6a1615e861534e6297e061f41e84c68c
I20260812 01:12:16.725574 2297 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a1615e861534e6297e061f41e84c68c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:16 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:38660
I20260812 01:12:16.725625 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a1615e861534e6297e061f41e84c68c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:16 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:48682
I20260812 01:12:16.725710 2805 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a1615e861534e6297e061f41e84c68c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:16 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:49322
I20260812 01:12:16.734393 3105 tablet_replica.cc:330] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab: stopping tablet replica
I20260812 01:12:16.734654 3105 raft_consensus.cc:2229] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:16.734918 3105 raft_consensus.cc:2258] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:16.737443 3105 ts_tablet_manager.cc:1905] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:16.738512 3107 tablet_replica.cc:330] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b: stopping tablet replica
I20260812 01:12:16.738722 3107 raft_consensus.cc:2229] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:16.738941 3107 raft_consensus.cc:2258] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:16.739518 3106 tablet_replica.cc:330] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e: stopping tablet replica
I20260812 01:12:16.739735 3106 raft_consensus.cc:2229] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:12:16.740388 3106 raft_consensus.cc:2258] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:16.741773 3107 ts_tablet_manager.cc:1905] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:16.745329 3105 ts_tablet_manager.cc:1918] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:12:16.745481 3105 log.cc:1196] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/6a1615e861534e6297e061f41e84c68c
I20260812 01:12:16.745927 3105 ts_tablet_manager.cc:1939] T 6a1615e861534e6297e061f41e84c68c P 82bb9adfa45144f4921bd16595f458ab: Deleting consensus metadata
I20260812 01:12:16.747115 2014 catalog_manager.cc:4841] TS 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229): tablet 6a1615e861534e6297e061f41e84c68c (table test-table [id=7e3198dcea954460bd926dc179bb40d2]) successfully deleted
I20260812 01:12:16.747403 3106 ts_tablet_manager.cc:1905] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:16.748664 3107 ts_tablet_manager.cc:1918] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:12:16.748822 3107 log.cc:1196] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/6a1615e861534e6297e061f41e84c68c
I20260812 01:12:16.749341 3107 ts_tablet_manager.cc:1939] T 6a1615e861534e6297e061f41e84c68c P 9fa7dff587e8483686bcc45c6102dd8b: Deleting consensus metadata
I20260812 01:12:16.750300 2018 catalog_manager.cc:4841] TS 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3:41959): tablet 6a1615e861534e6297e061f41e84c68c (table test-table [id=7e3198dcea954460bd926dc179bb40d2]) successfully deleted
I20260812 01:12:16.755162 3106 ts_tablet_manager.cc:1918] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:12:16.755466 3106 log.cc:1196] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/6a1615e861534e6297e061f41e84c68c
I20260812 01:12:16.755971 3106 ts_tablet_manager.cc:1939] T 6a1615e861534e6297e061f41e84c68c P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting consensus metadata
I20260812 01:12:16.757144 2013 catalog_manager.cc:4841] TS 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849): tablet 6a1615e861534e6297e061f41e84c68c (table test-table [id=7e3198dcea954460bd926dc179bb40d2]) successfully deleted
Aug 12 01:12:16 dist-test-slave-21b9 krb5kdc[1949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 01:12:16 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:16 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 01:12:16.819387 3118 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:16.806395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.20.151.62:34279)
0812 01:12:16.807207 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:16.807212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:16.807232 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:12:16.807418 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:16.807422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:16.807492 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:16.807590 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:16.809546 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.810374 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.812245 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.812698 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.812796 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.819006 (+ 6210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33036: BlockingRecv error: recv got EOF from 127.0.0.1:33036 (error 108)
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":582,"threads_started":1}
Aug 12 01:12:16 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM
I20260812 01:12:16.838075 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:16.822443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.20.151.62:34279)
0812 01:12:16.822643 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:16.822648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:16.822963 (+ 315us) server_negotiation.cc:407] Connection header received
0812 01:12:16.823138 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:16.823141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:16.823209 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:16.823329 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:16.824856 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.825650 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.827360 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:16.827782 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:16.827878 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:16.832543 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:16.832548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:16.832552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:16.832590 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:16.835367 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:16.836112 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:16.836118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:16.836120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:16.836265 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:16.836647 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:16.836650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:16.836652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:16.836957 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:16.837230 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:16.837516 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 01:12:16.837753 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 01:12:16.841935 2043 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:12:16.842542 2043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:12:16.850354 2297 tablet_service.cc:1467] Processing CreateTablet for tablet 77307f5baeb64e0c9a0bf94aa42f5b2b (DEFAULT_TABLE table=test-table [id=b709a6cf45ae4c139ad43658fd08002f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:16.850466 2805 tablet_service.cc:1467] Processing CreateTablet for tablet 77307f5baeb64e0c9a0bf94aa42f5b2b (DEFAULT_TABLE table=test-table [id=b709a6cf45ae4c139ad43658fd08002f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:16.850466 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 77307f5baeb64e0c9a0bf94aa42f5b2b (DEFAULT_TABLE table=test-table [id=b709a6cf45ae4c139ad43658fd08002f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:16.850858 2297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77307f5baeb64e0c9a0bf94aa42f5b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:16.850939 2805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77307f5baeb64e0c9a0bf94aa42f5b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:16.850986 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77307f5baeb64e0c9a0bf94aa42f5b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:16.863379 3131 tablet_bootstrap.cc:492] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b: Bootstrap starting.
I20260812 01:12:16.864173 3133 tablet_bootstrap.cc:492] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab: Bootstrap starting.
I20260812 01:12:16.865837 3132 tablet_bootstrap.cc:492] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: Bootstrap starting.
I20260812 01:12:16.866448 3133 tablet_bootstrap.cc:654] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:16.867372 3131 tablet_bootstrap.cc:654] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:16.868572 3132 tablet_bootstrap.cc:654] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:16.874989 3133 tablet_bootstrap.cc:492] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab: No bootstrap required, opened a new log
I20260812 01:12:16.875170 3133 ts_tablet_manager.cc:1397] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260812 01:12:16.876025 3133 raft_consensus.cc:348] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.876329 3133 raft_consensus.cc:374] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:16.876775 3133 raft_consensus.cc:729] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82bb9adfa45144f4921bd16595f458ab, State: Initialized, Role: FOLLOWER
I20260812 01:12:16.877136 3132 tablet_bootstrap.cc:492] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: No bootstrap required, opened a new log
I20260812 01:12:16.877112 3133 consensus_queue.cc:260] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.877300 3132 ts_tablet_manager.cc:1397] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260812 01:12:16.878222 3132 raft_consensus.cc:348] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.878599 3132 raft_consensus.cc:374] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:16.878728 3132 raft_consensus.cc:729] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d53f0dc4c74fbea95d45ea5474c78e, State: Initialized, Role: FOLLOWER
I20260812 01:12:16.879024 3131 tablet_bootstrap.cc:492] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b: No bootstrap required, opened a new log
I20260812 01:12:16.879251 3131 ts_tablet_manager.cc:1397] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260812 01:12:16.879091 3132 consensus_queue.cc:260] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.879755 3132 ts_tablet_manager.cc:1428] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 01:12:16.880744 3131 raft_consensus.cc:348] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.881172 3131 raft_consensus.cc:374] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:16.881331 3131 raft_consensus.cc:729] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa7dff587e8483686bcc45c6102dd8b, State: Initialized, Role: FOLLOWER
I20260812 01:12:16.881372 3133 ts_tablet_manager.cc:1428] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260812 01:12:16.882091 3131 consensus_queue.cc:260] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:16.882705 3131 ts_tablet_manager.cc:1428] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 01:12:17.103189 3002 raft_consensus.cc:482] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:12:17.103489 3002 raft_consensus.cc:504] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.104301 3002 leader_election.cc:288] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229), 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849)
I20260812 01:12:17.104950 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77307f5baeb64e0c9a0bf94aa42f5b2b" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82bb9adfa45144f4921bd16595f458ab" is_pre_election: true
I20260812 01:12:17.104919 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77307f5baeb64e0c9a0bf94aa42f5b2b" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" is_pre_election: true
I20260812 01:12:17.105247 2331 raft_consensus.cc:2454] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 0.
I20260812 01:12:17.105261 2583 raft_consensus.cc:2454] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 0.
I20260812 01:12:17.105767 2700 leader_election.cc:302] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d53f0dc4c74fbea95d45ea5474c78e, 9fa7dff587e8483686bcc45c6102dd8b; no voters:
I20260812 01:12:17.106041 3002 raft_consensus.cc:2781] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:12:17.106158 3002 raft_consensus.cc:482] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:12:17.106247 3002 raft_consensus.cc:3037] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:17.108614 3002 raft_consensus.cc:504] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.109228 3002 leader_election.cc:288] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 election: Requested vote from peers 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229), 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849)
I20260812 01:12:17.109777 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77307f5baeb64e0c9a0bf94aa42f5b2b" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82bb9adfa45144f4921bd16595f458ab"
I20260812 01:12:17.109817 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77307f5baeb64e0c9a0bf94aa42f5b2b" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d53f0dc4c74fbea95d45ea5474c78e"
I20260812 01:12:17.110031 2583 raft_consensus.cc:3037] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:17.110047 2331 raft_consensus.cc:3037] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:17.112387 2583 raft_consensus.cc:2454] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 1.
I20260812 01:12:17.112821 2700 leader_election.cc:302] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d53f0dc4c74fbea95d45ea5474c78e, 9fa7dff587e8483686bcc45c6102dd8b; no voters:
I20260812 01:12:17.113031 2331 raft_consensus.cc:2454] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 1.
I20260812 01:12:17.113102 3002 raft_consensus.cc:2781] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:17.118659 3002 raft_consensus.cc:686] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 1 LEADER]: Becoming Leader. State: Replica: 9fa7dff587e8483686bcc45c6102dd8b, State: Running, Role: LEADER
I20260812 01:12:17.119177 3002 consensus_queue.cc:237] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.122299 2040 catalog_manager.cc:5495] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3). New cstate: current_term: 1 leader_uuid: "9fa7dff587e8483686bcc45c6102dd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } health_report { overall_health: HEALTHY } } }
I20260812 01:12:17.165208 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:17.158872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.20.151.3:41959)
0812 01:12:17.159766 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:17.159773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:17.159807 (+ 34us) server_negotiation.cc:407] Connection header received
0812 01:12:17.159880 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:17.159886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:17.159982 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:17.160084 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:12:17.160429 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.161610 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.163160 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.163867 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.163993 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:17.164441 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:12:17.164580 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:12:17.164661 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:17.164761 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 01:12:17.164841 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":592,"threads_started":1}
I20260812 01:12:17.168063 2583 raft_consensus.cc:1261] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Refusing update from remote peer 9fa7dff587e8483686bcc45c6102dd8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:17.168119 2331 raft_consensus.cc:1261] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Refusing update from remote peer 9fa7dff587e8483686bcc45c6102dd8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:17.168761 3142 consensus_queue.cc:1035] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:17.169101 3002 consensus_queue.cc:1035] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:17.181272 2043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:17.181566 2043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:17.185101 2043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8: Sending DeleteTablet for 3 replicas of tablet 77307f5baeb64e0c9a0bf94aa42f5b2b
I20260812 01:12:17.185894 2297 tablet_service.cc:1514] Processing DeleteTablet for tablet 77307f5baeb64e0c9a0bf94aa42f5b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:17 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:38660
I20260812 01:12:17.186097 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 77307f5baeb64e0c9a0bf94aa42f5b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:17 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:48682
I20260812 01:12:17.186196 3105 tablet_replica.cc:330] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab: stopping tablet replica
I20260812 01:12:17.186394 3105 raft_consensus.cc:2229] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:17.186420 3106 tablet_replica.cc:330] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: stopping tablet replica
I20260812 01:12:17.186420 2805 tablet_service.cc:1514] Processing DeleteTablet for tablet 77307f5baeb64e0c9a0bf94aa42f5b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:17 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:49322
I20260812 01:12:17.186609 3105 raft_consensus.cc:2258] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:17.186695 3107 tablet_replica.cc:330] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b: stopping tablet replica
I20260812 01:12:17.186868 3107 raft_consensus.cc:2229] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:12:17.187165 3107 raft_consensus.cc:2258] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:17.187772 3105 ts_tablet_manager.cc:1905] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:17.188594 3107 ts_tablet_manager.cc:1905] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:17.191774 2043 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:12:17.192473 2043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:12:17.194779 3107 ts_tablet_manager.cc:1918] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:17.194940 3107 log.cc:1196] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/77307f5baeb64e0c9a0bf94aa42f5b2b
I20260812 01:12:17.195375 3107 ts_tablet_manager.cc:1939] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 9fa7dff587e8483686bcc45c6102dd8b: Deleting consensus metadata
I20260812 01:12:17.195700 3106 raft_consensus.cc:2229] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:17.196403 3106 pending_rounds.cc:62] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: Trying to abort 2 pending ops.
I20260812 01:12:17.196448 2018 catalog_manager.cc:4841] TS 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3:41959): tablet 77307f5baeb64e0c9a0bf94aa42f5b2b (table test-table [id=b709a6cf45ae4c139ad43658fd08002f]) successfully deleted
I20260812 01:12:17.196553 3106 pending_rounds.cc:68] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7317492273834389504 op_type: WRITE_OP write_request { tablet_id: "77307f5baeb64e0c9a0bf94aa42f5b2b" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\235\223\357\323\006\"8\n\ntest-admin\022*\n b709a6cf45ae4c139ad43658fd08002f\020\001\030\001 \001(\001" signature: "<redacted>"",(\270o\247\371fY\373d\202S<\355\373\025E\016\212+I5\276H\021\255\353|,\014\377\200\346\207\n\360\033@.\367\371\273\\\240\247\351i\200s\304m\276sx\243$\357\206\007a\231V0\232" signing_key_seq_num: 0 } } request_id { client_id: "db2a9744fba14e05b3d3dc7d4d2b7d57" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260812 01:12:17.197309 3106 pending_rounds.cc:68] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317492273642819584 op_type: NO_OP noop_request { }
I20260812 01:12:17.197485 3106 raft_consensus.cc:2866] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260812 01:12:17.197624 3106 raft_consensus.cc:2258] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:17.198607 3105 ts_tablet_manager.cc:1918] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:17.198786 3105 log.cc:1196] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/77307f5baeb64e0c9a0bf94aa42f5b2b
I20260812 01:12:17.199250 3105 ts_tablet_manager.cc:1939] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 82bb9adfa45144f4921bd16595f458ab: Deleting consensus metadata
W20260812 01:12:17.199710 3151 consensus_queue.cc:1159] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260812 01:12:17.200773 2297 tablet_service.cc:1467] Processing CreateTablet for tablet 988af89d363141ac9846d4ddf0d4443b (DEFAULT_TABLE table=test-table [id=3658adef138645e7b4238ecdfb63bc82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:17.201013 3106 ts_tablet_manager.cc:1905] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:17.201138 2297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988af89d363141ac9846d4ddf0d4443b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:17.203418 2014 catalog_manager.cc:4841] TS 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229): tablet 77307f5baeb64e0c9a0bf94aa42f5b2b (table test-table [id=b709a6cf45ae4c139ad43658fd08002f]) successfully deleted
I20260812 01:12:17.204344 2805 tablet_service.cc:1467] Processing CreateTablet for tablet 988af89d363141ac9846d4ddf0d4443b (DEFAULT_TABLE table=test-table [id=3658adef138645e7b4238ecdfb63bc82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:17.204773 2805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988af89d363141ac9846d4ddf0d4443b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:17.206995 3133 tablet_bootstrap.cc:492] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab: Bootstrap starting.
I20260812 01:12:17.208645 3133 tablet_bootstrap.cc:654] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:17.209569 3131 tablet_bootstrap.cc:492] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b: Bootstrap starting.
I20260812 01:12:17.211035 3131 tablet_bootstrap.cc:654] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:17.212873 3131 tablet_bootstrap.cc:492] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b: No bootstrap required, opened a new log
I20260812 01:12:17.213030 3131 ts_tablet_manager.cc:1397] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 01:12:17.213783 3131 raft_consensus.cc:348] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.213981 3131 raft_consensus.cc:374] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:17.214063 3131 raft_consensus.cc:729] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa7dff587e8483686bcc45c6102dd8b, State: Initialized, Role: FOLLOWER
I20260812 01:12:17.214321 3131 consensus_queue.cc:260] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.214905 3131 ts_tablet_manager.cc:1428] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 01:12:17.216311 3133 tablet_bootstrap.cc:492] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab: No bootstrap required, opened a new log
I20260812 01:12:17.216502 3133 ts_tablet_manager.cc:1397] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260812 01:12:17.216787 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 988af89d363141ac9846d4ddf0d4443b (DEFAULT_TABLE table=test-table [id=3658adef138645e7b4238ecdfb63bc82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:17.217249 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988af89d363141ac9846d4ddf0d4443b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:17.217271 3133 raft_consensus.cc:348] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.217504 3133 raft_consensus.cc:374] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:17.217592 3133 raft_consensus.cc:729] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82bb9adfa45144f4921bd16595f458ab, State: Initialized, Role: FOLLOWER
I20260812 01:12:17.217830 3133 consensus_queue.cc:260] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.218449 3133 ts_tablet_manager.cc:1428] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 01:12:17.220309 3106 ts_tablet_manager.cc:1918] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:17.220541 3106 log.cc:1196] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/77307f5baeb64e0c9a0bf94aa42f5b2b
I20260812 01:12:17.221081 3106 ts_tablet_manager.cc:1939] T 77307f5baeb64e0c9a0bf94aa42f5b2b P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting consensus metadata
I20260812 01:12:17.222496 2013 catalog_manager.cc:4841] TS 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849): tablet 77307f5baeb64e0c9a0bf94aa42f5b2b (table test-table [id=b709a6cf45ae4c139ad43658fd08002f]) successfully deleted
I20260812 01:12:17.226017 3132 tablet_bootstrap.cc:492] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e: Bootstrap starting.
I20260812 01:12:17.227881 3132 tablet_bootstrap.cc:654] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:17.229725 3132 tablet_bootstrap.cc:492] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e: No bootstrap required, opened a new log
I20260812 01:12:17.229874 3132 ts_tablet_manager.cc:1397] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260812 01:12:17.230551 3132 raft_consensus.cc:348] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.230793 3132 raft_consensus.cc:374] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:17.230902 3132 raft_consensus.cc:729] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d53f0dc4c74fbea95d45ea5474c78e, State: Initialized, Role: FOLLOWER
I20260812 01:12:17.231220 3132 consensus_queue.cc:260] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.231884 3132 ts_tablet_manager.cc:1428] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 01:12:17.362130 3142 raft_consensus.cc:482] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:12:17.362406 3142 raft_consensus.cc:504] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.363220 3142 leader_election.cc:288] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229), 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849)
I20260812 01:12:17.363942 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988af89d363141ac9846d4ddf0d4443b" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82bb9adfa45144f4921bd16595f458ab" is_pre_election: true
I20260812 01:12:17.364008 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988af89d363141ac9846d4ddf0d4443b" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" is_pre_election: true
I20260812 01:12:17.364220 2331 raft_consensus.cc:2454] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 0.
I20260812 01:12:17.364271 2583 raft_consensus.cc:2454] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 0.
I20260812 01:12:17.364686 2701 leader_election.cc:302] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82bb9adfa45144f4921bd16595f458ab, 9fa7dff587e8483686bcc45c6102dd8b; no voters:
I20260812 01:12:17.365033 3142 raft_consensus.cc:2781] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:12:17.365144 3142 raft_consensus.cc:482] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:12:17.365250 3142 raft_consensus.cc:3037] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:17.367457 3142 raft_consensus.cc:504] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.368178 3142 leader_election.cc:288] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 election: Requested vote from peers 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229), 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849)
I20260812 01:12:17.368680 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988af89d363141ac9846d4ddf0d4443b" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82bb9adfa45144f4921bd16595f458ab"
I20260812 01:12:17.368772 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988af89d363141ac9846d4ddf0d4443b" candidate_uuid: "9fa7dff587e8483686bcc45c6102dd8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d53f0dc4c74fbea95d45ea5474c78e"
I20260812 01:12:17.368955 2331 raft_consensus.cc:3037] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:17.369025 2583 raft_consensus.cc:3037] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:17.372063 2331 raft_consensus.cc:2454] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 1.
I20260812 01:12:17.372087 2583 raft_consensus.cc:2454] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fa7dff587e8483686bcc45c6102dd8b in term 1.
I20260812 01:12:17.372534 2701 leader_election.cc:302] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82bb9adfa45144f4921bd16595f458ab, 9fa7dff587e8483686bcc45c6102dd8b; no voters:
I20260812 01:12:17.373155 3142 raft_consensus.cc:2781] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:17.373318 3142 raft_consensus.cc:686] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 1 LEADER]: Becoming Leader. State: Replica: 9fa7dff587e8483686bcc45c6102dd8b, State: Running, Role: LEADER
I20260812 01:12:17.373600 3142 consensus_queue.cc:237] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } }
I20260812 01:12:17.376492 2043 catalog_manager.cc:5495] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3). New cstate: current_term: 1 leader_uuid: "9fa7dff587e8483686bcc45c6102dd8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fa7dff587e8483686bcc45c6102dd8b" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41959 } health_report { overall_health: HEALTHY } } }
I20260812 01:12:17.434648 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:17.428853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:58233 (local address 127.20.151.2:34849)
0812 01:12:17.429433 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:17.429439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:17.429792 (+ 353us) server_negotiation.cc:407] Connection header received
0812 01:12:17.429980 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:17.429984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:17.430051 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:17.430134 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:12:17.430639 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.431455 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.432885 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.433298 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.433406 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:17.433832 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:12:17.433940 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:12:17.434016 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:17.434158 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 01:12:17.434239 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":278,"threads_started":1}
I20260812 01:12:17.434631 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:17.428872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:34849 (local address 127.20.151.3:58233)
0812 01:12:17.429689 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:17.429704 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:17.429849 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:17.430166 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:17.430170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:17.430183 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 01:12:17.430464 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:17.430474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.431579 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.431585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:17.432745 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:17.432758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.433447 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.433454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:17.433578 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:17.433638 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 01:12:17.434032 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 01:12:17.434045 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 01:12:17.434120 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 01:12:17.434264 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":474,"threads_started":1}
I20260812 01:12:17.442552 2583 raft_consensus.cc:1261] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Refusing update from remote peer 9fa7dff587e8483686bcc45c6102dd8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:17.442596 2331 raft_consensus.cc:1261] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Refusing update from remote peer 9fa7dff587e8483686bcc45c6102dd8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:17.443172 3002 consensus_queue.cc:1035] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d53f0dc4c74fbea95d45ea5474c78e" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 34849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:17.443482 3142 consensus_queue.cc:1035] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "82bb9adfa45144f4921bd16595f458ab" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 38229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:17.473872 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:17.467465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:44433 (local address 127.20.151.3:41959)
0812 01:12:17.467861 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:17.467867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:17.468336 (+ 469us) server_negotiation.cc:407] Connection header received
0812 01:12:17.468556 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:17.468561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:17.468649 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:17.468752 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:12:17.469532 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.470289 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.472207 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.472780 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.472915 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:17.473305 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:12:17.473418 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:12:17.473492 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:17.473575 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 01:12:17.473649 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260812 01:12:17.473964 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:17.467506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:41959 (local address 127.20.151.2:44433)
0812 01:12:17.468237 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:17.468260 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:17.468409 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:17.468758 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:17.468764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:17.468777 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 01:12:17.469334 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:17.469352 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.470396 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.470402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:17.472072 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:17.472086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.472973 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.472980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:17.473135 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:17.473177 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 01:12:17.473497 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 01:12:17.473506 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 01:12:17.473561 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 01:12:17.473692 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":373,"threads_started":1}
I20260812 01:12:17.505100 2043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:17.505334 2043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:17.508067 2043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e7d83789b294c80b971f67fe4151dd8: Sending DeleteTablet for 3 replicas of tablet 988af89d363141ac9846d4ddf0d4443b
I20260812 01:12:17.508898 2297 tablet_service.cc:1514] Processing DeleteTablet for tablet 988af89d363141ac9846d4ddf0d4443b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:17 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:38660
I20260812 01:12:17.509055 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 988af89d363141ac9846d4ddf0d4443b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:17 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:48682
I20260812 01:12:17.509189 2805 tablet_service.cc:1514] Processing DeleteTablet for tablet 988af89d363141ac9846d4ddf0d4443b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:17 UTC) from {username='oryx', principal='oryx/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:49322
I20260812 01:12:17.509254 3105 tablet_replica.cc:330] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab: stopping tablet replica
I20260812 01:12:17.509388 3106 tablet_replica.cc:330] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e: stopping tablet replica
I20260812 01:12:17.509479 3107 tablet_replica.cc:330] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b: stopping tablet replica
I20260812 01:12:17.509451 3105 raft_consensus.cc:2229] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:17.509563 3106 raft_consensus.cc:2229] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:17.509631 3107 raft_consensus.cc:2229] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:12:17.509688 3105 raft_consensus.cc:2258] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:17.509781 3106 raft_consensus.cc:2258] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:17.509976 3107 raft_consensus.cc:2258] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:17.510553 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 2181
I20260812 01:12:17.510870 3106 ts_tablet_manager.cc:1905] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:17.512574 3105 ts_tablet_manager.cc:1905] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:17.512749 3107 ts_tablet_manager.cc:1905] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:17.518126 3106 ts_tablet_manager.cc:1918] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:12:17.518271 3106 log.cc:1196] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/988af89d363141ac9846d4ddf0d4443b
I20260812 01:12:17.518725 3106 ts_tablet_manager.cc:1939] T 988af89d363141ac9846d4ddf0d4443b P 64d53f0dc4c74fbea95d45ea5474c78e: Deleting consensus metadata
I20260812 01:12:17.518972 3105 ts_tablet_manager.cc:1918] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:12:17.519119 3105 log.cc:1196] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/988af89d363141ac9846d4ddf0d4443b
I20260812 01:12:17.519932 2013 catalog_manager.cc:4841] TS 64d53f0dc4c74fbea95d45ea5474c78e (127.20.151.2:34849): tablet 988af89d363141ac9846d4ddf0d4443b (table test-table [id=3658adef138645e7b4238ecdfb63bc82]) successfully deleted
I20260812 01:12:17.519997 3107 ts_tablet_manager.cc:1918] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 01:12:17.520121 3107 log.cc:1196] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/988af89d363141ac9846d4ddf0d4443b
I20260812 01:12:17.520542 3107 ts_tablet_manager.cc:1939] T 988af89d363141ac9846d4ddf0d4443b P 9fa7dff587e8483686bcc45c6102dd8b: Deleting consensus metadata
I20260812 01:12:17.521677 2018 catalog_manager.cc:4841] TS 9fa7dff587e8483686bcc45c6102dd8b (127.20.151.3:41959): tablet 988af89d363141ac9846d4ddf0d4443b (table test-table [id=3658adef138645e7b4238ecdfb63bc82]) successfully deleted
I20260812 01:12:17.522049 3105 ts_tablet_manager.cc:1939] T 988af89d363141ac9846d4ddf0d4443b P 82bb9adfa45144f4921bd16595f458ab: Deleting consensus metadata
I20260812 01:12:17.522981 2014 catalog_manager.cc:4841] TS 82bb9adfa45144f4921bd16595f458ab (127.20.151.1:38229): tablet 988af89d363141ac9846d4ddf0d4443b (table test-table [id=3658adef138645e7b4238ecdfb63bc82]) successfully deleted
Aug 12 01:12:17 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.151.1@KRBTEST.COM
I20260812 01:12:17.535629 3181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:17.511847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.20.151.1:38229)
0812 01:12:17.521921 (+ 10074us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:17.521927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:17.521948 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:12:17.522007 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:17.522012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:17.522097 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:17.522214 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:17.523518 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.524311 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.525512 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.525910 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.526018 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:17.529846 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:17.529863 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:17.529881 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:17.529918 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:17.532824 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:17.533479 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:17.533486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:17.533489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:17.533643 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:17.534005 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:17.534008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:17.534010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:17.534392 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:17.534678 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:17.535010 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 01:12:17.535292 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9914,"thread_start_us":9790,"threads_started":1}
I20260812 01:12:17.536332 2409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:17.636371 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 2181
I20260812 01:12:17.668691 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 2424
Aug 12 01:12:17 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.151.2@KRBTEST.COM
I20260812 01:12:17.682986 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:17.669506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.20.151.2:34849)
0812 01:12:17.669730 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:17.669736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:17.669755 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:12:17.669883 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:17.669887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:17.669953 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:17.670035 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:17.671210 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.671961 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.673283 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.673711 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.673845 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:17.677775 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:17.677789 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:17.677807 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:17.677847 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:17.680616 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:17.681246 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:17.681252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:17.681254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:17.681393 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:17.681672 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:17.681675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:17.681677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:17.682026 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:17.682249 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:17.682499 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 01:12:17.682676 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 01:12:17.683827 2660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:17.837522 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 2424
I20260812 01:12:17.871840 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 2675
Aug 12 01:12:17 dist-test-slave-21b9 krb5kdc[1949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.151.3@KRBTEST.COM
I20260812 01:12:17.891578 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:17.872627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42678 (local address 127.20.151.3:41959)
0812 01:12:17.872850 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:17.872857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:17.872928 (+ 71us) server_negotiation.cc:407] Connection header received
0812 01:12:17.873091 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:17.873095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:17.873197 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:17.873280 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:17.874560 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.875334 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.876596 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:17.876988 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:17.877074 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:17.885806 (+ 8732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:17.885846 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:17.885865 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:17.885911 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:17.888697 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:17.889433 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:17.889440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:17.889443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:17.889603 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:17.889970 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:17.889974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:17.889976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:17.890356 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:17.890667 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:17.890983 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 01:12:17.891219 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 01:12:17.892244 2915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:18.023761 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 2675
I20260812 01:12:18.053427 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 1979
I20260812 01:12:18.054734 2117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:18.175243 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 1979
2026-08-12T01:12:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7471 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:12:18 dist-test-slave-21b9 krb5kdc[3192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:12:18 dist-test-slave-21b9 krb5kdc[3192](info): set up 2 sockets
Aug 12 01:12:18 dist-test-slave-21b9 krb5kdc[3192](info): commencing operation
krb5kdc: starting...
W20260812 01:12:19.450729 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:12:19 dist-test-slave-21b9 krb5kdc[3192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:12:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:12:19.748205 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:39583
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:39583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:20.095901 3210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:20.096210 3210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:20.096316 3210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:20.103582 3210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:12:20.103718 3210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:20.103770 3210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:20.103826 3210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:12:20.103874 3210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:12:20.114030 3210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:39583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:39583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:20.115370 3210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:20.116779 3210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:20.129200 3219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:20.129264 3228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:20.129228 3223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:20.129743 3210 server_base.cc:1034] running on GCE node
I20260812 01:12:20.130594 3210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:20.132167 3210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:20.133400 3210 hybrid_clock.cc:648] HybridClock initialized: now 1786497140133332 us; error 43 us; skew 500 ppm
I20260812 01:12:20.133790 3210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T01:12:20Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2012 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:12:20 dist-test-slave-21b9 krb5kdc[3237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:12:20 dist-test-slave-21b9 krb5kdc[3237](info): set up 2 sockets
Aug 12 01:12:20 dist-test-slave-21b9 krb5kdc[3237](info): commencing operation
krb5kdc: starting...
W20260812 01:12:21.476336 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.002s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:12:21 dist-test-slave-21b9 krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:12:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:12:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:12:21.781308 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:39533
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:39533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:22.129618 3260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:22.129954 3260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:22.130110 3260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:22.137567 3260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:12:22.137712 3260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:22.137796 3260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:22.137874 3260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:12:22.137949 3260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:12:22.148252 3260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:39533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:39533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:22.149696 3260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:22.151075 3260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:22.163084 3285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:22.163148 3274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:22.163177 3279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:22.163868 3260 server_base.cc:1034] running on GCE node
I20260812 01:12:22.164752 3260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:22.166216 3260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:22.167431 3260 hybrid_clock.cc:648] HybridClock initialized: now 1786497142167377 us; error 43 us; skew 500 ppm
I20260812 01:12:22.167798 3260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:22 dist-test-slave-21b9 krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497142, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:22.171051 3260 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:12:22.172716 3260 webserver.cc:460] Webserver started at http://127.20.151.62:37969/ using document root <none> and password file <none>
I20260812 01:12:22.173425 3260 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:22.173547 3260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:22.173863 3260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:22.176999 3260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "16eb47f310b444c8805ae07a75547fc7"
format_stamp: "Formatted at 2026-08-12 01:12:22 on dist-test-slave-21b9"
server_key: "dc769349e420af52f38ba66d6c10422a"
server_key_iv: "bf3325b3d4198daaf968cb06633b48b3"
server_key_version: "encryptionkey@0"
I20260812 01:12:22.177767 3260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "16eb47f310b444c8805ae07a75547fc7"
format_stamp: "Formatted at 2026-08-12 01:12:22 on dist-test-slave-21b9"
server_key: "dc769349e420af52f38ba66d6c10422a"
server_key_iv: "bf3325b3d4198daaf968cb06633b48b3"
server_key_version: "encryptionkey@0"
I20260812 01:12:22.183074 3260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:12:22.186671 3295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:22.188165 3260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:12:22.188359 3260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "16eb47f310b444c8805ae07a75547fc7"
format_stamp: "Formatted at 2026-08-12 01:12:22 on dist-test-slave-21b9"
server_key: "dc769349e420af52f38ba66d6c10422a"
server_key_iv: "bf3325b3d4198daaf968cb06633b48b3"
server_key_version: "encryptionkey@0"
I20260812 01:12:22.188587 3260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:22.210665 3260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:22.215487 3260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:22.215905 3260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:22.234596 3260 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:39533
I20260812 01:12:22.234606 3384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:39533 every 8 connection(s)
I20260812 01:12:22.236155 3260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:12:22.240103 3385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:22.242043 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 3260
I20260812 01:12:22.242297 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:12:22.243144 21084 external_mini_cluster.cc:1428] Setting key f65cb963ce0a8578d9a18c47463a6800
I20260812 01:12:22.253235 3385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7: Bootstrap starting.
I20260812 01:12:22.257170 3385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:22.258432 3385 log.cc:824] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7: Log is configured to *not* fsync() on all Append() calls
Aug 12 01:12:22 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:12:22.262058 3385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7: No bootstrap required, opened a new log
I20260812 01:12:22.268404 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:22.245444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.20.151.62:39533)
0812 01:12:22.246279 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:22.246291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:22.246336 (+ 45us) server_negotiation.cc:407] Connection header received
0812 01:12:22.248275 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:22.248301 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:22.248717 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:22.249034 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:22.250894 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:22.252317 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:22.254056 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:22.254766 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:22.254915 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:22.259766 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:22.259775 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:22.259790 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:22.259832 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:22.263718 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:22.264411 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:22.264420 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:22.264434 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:22.264650 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:22.264993 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:22.264998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:22.265002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:22.265756 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:22.266083 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:22.266429 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 01:12:22.266890 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":356,"threads_started":1}
I20260812 01:12:22.271833 3385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } }
I20260812 01:12:22.272208 3385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:22.272286 3385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eb47f310b444c8805ae07a75547fc7, State: Initialized, Role: FOLLOWER
I20260812 01:12:22.272797 3385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } }
I20260812 01:12:22.273012 3385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:12:22.273109 3385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:12:22.273228 3385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:22.275771 3385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } }
I20260812 01:12:22.276304 3385 leader_election.cc:302] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16eb47f310b444c8805ae07a75547fc7; no voters:
I20260812 01:12:22.276976 3385 leader_election.cc:288] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:12:22.277065 3399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:22.277760 3399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 16eb47f310b444c8805ae07a75547fc7, State: Running, Role: LEADER
I20260812 01:12:22.278308 3399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } }
I20260812 01:12:22.279119 3385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:12:22.281041 3406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16eb47f310b444c8805ae07a75547fc7. Latest consensus state: current_term: 1 leader_uuid: "16eb47f310b444c8805ae07a75547fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } } }
I20260812 01:12:22.281016 3401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16eb47f310b444c8805ae07a75547fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } } }
I20260812 01:12:22.281461 3401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:22.281461 3406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:22.287642 3415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:12:22.292878 3415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:12:22.302595 3415 catalog_manager.cc:1347] Generated new cluster ID: f7ca5d437e2340b0938152f4d086e0e2
I20260812 01:12:22.302739 3415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:12:22.313804 3415 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:12:22.315878 3415 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:12:22.330118 3415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7: Generated new TSK 0
I20260812 01:12:22.331328 3415 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:12:22.402812 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39533
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:22.760514 3437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:22.760823 3437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:22.760967 3437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:22.768498 3437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:22.768633 3437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:22.768815 3437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:12:22.779054 3437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:22.780640 3437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:22.781993 3437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:22.795922 3446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:22.795979 3456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:22.796036 3448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:22.797070 3437 server_base.cc:1034] running on GCE node
I20260812 01:12:22.797700 3437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:22.798774 3437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:22.800020 3437 hybrid_clock.cc:648] HybridClock initialized: now 1786497142799939 us; error 63 us; skew 500 ppm
I20260812 01:12:22.800369 3437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:22 dist-test-slave-21b9 krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497142, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:22.803884 3437 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:12:22.805552 3437 webserver.cc:460] Webserver started at http://127.20.151.1:45201/ using document root <none> and password file <none>
I20260812 01:12:22.806344 3437 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:22.806471 3437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:22.806802 3437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:22.809909 3437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "642555a60fa345daa6d76a4489dc155f"
format_stamp: "Formatted at 2026-08-12 01:12:22 on dist-test-slave-21b9"
server_key: "2533204cde96467e7da1b6af97772068"
server_key_iv: "0d88466886ca8c12da781d03eb045521"
server_key_version: "encryptionkey@0"
I20260812 01:12:22.810696 3437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "642555a60fa345daa6d76a4489dc155f"
format_stamp: "Formatted at 2026-08-12 01:12:22 on dist-test-slave-21b9"
server_key: "2533204cde96467e7da1b6af97772068"
server_key_iv: "0d88466886ca8c12da781d03eb045521"
server_key_version: "encryptionkey@0"
I20260812 01:12:22.816241 3437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 01:12:22.820209 3470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:22.821524 3437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:12:22.821686 3437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "642555a60fa345daa6d76a4489dc155f"
format_stamp: "Formatted at 2026-08-12 01:12:22 on dist-test-slave-21b9"
server_key: "2533204cde96467e7da1b6af97772068"
server_key_iv: "0d88466886ca8c12da781d03eb045521"
server_key_version: "encryptionkey@0"
I20260812 01:12:22.821877 3437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:22.855751 3437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:22.860334 3437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:22.860668 3437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:22.861719 3437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:22.863337 3437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:22.863425 3437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:22.863575 3437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:22.863632 3437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:22.896453 3437 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:41571
I20260812 01:12:22.896510 3672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:41571 every 8 connection(s)
I20260812 01:12:22.897719 3437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:12:22.906226 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 3437
I20260812 01:12:22.906430 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:12:22.907075 21084 external_mini_cluster.cc:1428] Setting key 0f190a66f4bc6c54578b9c85bd5d0a42
Aug 12 01:12:22 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497142, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:12:22.923238 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:22.900244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:39877 (local address 127.20.151.62:39533)
0812 01:12:22.900778 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:22.900785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:22.901920 (+ 1135us) server_negotiation.cc:407] Connection header received
0812 01:12:22.903851 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:22.903856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:22.903957 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:22.904060 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:22.905586 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:22.906312 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:22.908153 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:22.908600 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:22.908718 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:22.916570 (+ 7852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:22.916578 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:22.916582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:22.916630 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:22.919743 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:22.920575 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:22.920581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:22.920584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:22.920740 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:22.921208 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:22.921212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:22.921215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:22.921553 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:22.921695 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:22.922754 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0812 01:12:22.922958 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":276,"threads_started":1}
I20260812 01:12:22.924602 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:22.900840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:39877)
0812 01:12:22.901745 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:22.901791 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:22.903587 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:22.904308 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:22.904316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:22.904819 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:22.905411 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:22.905430 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:22.906515 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:22.906520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:22.908024 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:22.908038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:22.908840 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:22.908847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:22.909175 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:22.913858 (+ 4683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:22.913891 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:22.916277 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:22.919914 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:22.919923 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:22.919938 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:22.920427 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:22.920869 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:22.920873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:22.920875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:22.921094 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:22.921706 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:22.921713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:22.922215 (+ 502us) client_negotiation.cc:769] Sending connection context
0812 01:12:22.922557 (+ 342us) client_negotiation.cc:240] Negotiation successful
0812 01:12:22.923191 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":370,"threads_started":1}
I20260812 01:12:22.926383 3674 heartbeater.cc:347] Connected to a master server at 127.20.151.62:39533
I20260812 01:12:22.926781 3674 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:22.927584 3674 heartbeater.cc:511] Master 127.20.151.62:39533 requested a full tablet report, sending...
I20260812 01:12:22.929721 3320 ts_manager.cc:194] Registered new tserver with Master: 642555a60fa345daa6d76a4489dc155f (127.20.151.1:41571)
I20260812 01:12:22.931303 3320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:39877
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:12:22.975632 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39533
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:23.328966 3688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:23.329288 3688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:23.329428 3688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:23.337000 3688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:23.337119 3688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:23.337263 3688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:12:23.347213 3688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:23.348574 3688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:23.349931 3688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:23.363353 3694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:23.363251 3695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:23.363394 3698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:23.364740 3688 server_base.cc:1034] running on GCE node
I20260812 01:12:23.365453 3688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:23.366492 3688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:23.367707 3688 hybrid_clock.cc:648] HybridClock initialized: now 1786497143367660 us; error 38 us; skew 500 ppm
I20260812 01:12:23.368044 3688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:23 dist-test-slave-21b9 krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497143, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:23.371881 3688 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:12:23.373453 3688 webserver.cc:460] Webserver started at http://127.20.151.2:44747/ using document root <none> and password file <none>
I20260812 01:12:23.374244 3688 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:23.374363 3688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:23.374691 3688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:23.377861 3688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "55b812d1684243f69e774dfbd8d273ea"
format_stamp: "Formatted at 2026-08-12 01:12:23 on dist-test-slave-21b9"
server_key: "704a3f04ed74dc083f1426e333fd760e"
server_key_iv: "11c7d2fd8b7ea4122e0803ea1b1a4497"
server_key_version: "encryptionkey@0"
I20260812 01:12:23.378684 3688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "55b812d1684243f69e774dfbd8d273ea"
format_stamp: "Formatted at 2026-08-12 01:12:23 on dist-test-slave-21b9"
server_key: "704a3f04ed74dc083f1426e333fd760e"
server_key_iv: "11c7d2fd8b7ea4122e0803ea1b1a4497"
server_key_version: "encryptionkey@0"
I20260812 01:12:23.384101 3688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260812 01:12:23.387671 3707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:23.389232 3688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:12:23.389406 3688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "55b812d1684243f69e774dfbd8d273ea"
format_stamp: "Formatted at 2026-08-12 01:12:23 on dist-test-slave-21b9"
server_key: "704a3f04ed74dc083f1426e333fd760e"
server_key_iv: "11c7d2fd8b7ea4122e0803ea1b1a4497"
server_key_version: "encryptionkey@0"
I20260812 01:12:23.389623 3688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:23.405264 3688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:23.409802 3688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:23.410099 3688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:23.411147 3688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:23.412634 3688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:23.412711 3688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:23.412838 3688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:23.412889 3688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:23.444991 3688 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:39511
I20260812 01:12:23.445056 3878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:39511 every 8 connection(s)
I20260812 01:12:23.446326 3688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:12:23.456140 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 3688
I20260812 01:12:23.456368 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:12:23.456918 21084 external_mini_cluster.cc:1428] Setting key 5a60152ec75ef622153e0cc919d75c24
Aug 12 01:12:23 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497143, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:12:23.468467 3884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:23.449047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:60623 (local address 127.20.151.62:39533)
0812 01:12:23.449544 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:23.449551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:23.450579 (+ 1028us) server_negotiation.cc:407] Connection header received
0812 01:12:23.452474 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:23.452479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:23.452551 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:23.452642 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:23.454239 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:23.455003 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:23.457299 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:23.457778 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:23.457916 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:23.462339 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:23.462344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:23.462348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:23.462386 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:23.465111 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:23.466028 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:23.466032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:23.466034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:23.466168 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:23.466634 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:23.466637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:23.466639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:23.466906 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:23.467030 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:23.467964 (+ 934us) server_negotiation.cc:299] Negotiation successful
0812 01:12:23.468184 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":233,"threads_started":1}
I20260812 01:12:23.469844 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:23.449606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:60623)
0812 01:12:23.450381 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:23.450419 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:23.452197 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:23.452877 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:23.452885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:23.453464 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:23.454062 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:23.454082 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:23.455210 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:23.455215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:23.457140 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:23.457158 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:23.458042 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:23.458049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:23.458366 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:23.459225 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:23.459256 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:23.462016 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:23.465271 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:23.465282 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:23.465302 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:23.465910 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:23.466269 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:23.466273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:23.466276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:23.466539 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:23.467025 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:23.467034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:23.467526 (+ 492us) client_negotiation.cc:769] Sending connection context
0812 01:12:23.467846 (+ 320us) client_negotiation.cc:240] Negotiation successful
0812 01:12:23.468431 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":297,"threads_started":1}
I20260812 01:12:23.471467 3879 heartbeater.cc:347] Connected to a master server at 127.20.151.62:39533
I20260812 01:12:23.471804 3879 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:23.472579 3879 heartbeater.cc:511] Master 127.20.151.62:39533 requested a full tablet report, sending...
I20260812 01:12:23.474076 3320 ts_manager.cc:194] Registered new tserver with Master: 55b812d1684243f69e774dfbd8d273ea (127.20.151.2:39511)
I20260812 01:12:23.474918 3320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:60623
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:12:23.528784 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39533
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:23.878351 3889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:23.878654 3889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:23.878778 3889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:23.886361 3889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:23.886498 3889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:23.886658 3889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:12:23.897002 3889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:23.898299 3889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:23.899652 3889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:23.912727 3902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:23.912770 3896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:23.912796 3895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:23.913694 3889 server_base.cc:1034] running on GCE node
I20260812 01:12:23.914304 3889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:23.915355 3889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:23.916596 3889 hybrid_clock.cc:648] HybridClock initialized: now 1786497143916544 us; error 41 us; skew 500 ppm
I20260812 01:12:23.916931 3889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:23 dist-test-slave-21b9 krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497143, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:23.920212 3889 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:12:23.921696 3889 webserver.cc:460] Webserver started at http://127.20.151.3:46097/ using document root <none> and password file <none>
I20260812 01:12:23.922363 3889 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:23.922464 3889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:23.922746 3889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:23.925848 3889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "4ef55a1485b54be696856446dface8f9"
format_stamp: "Formatted at 2026-08-12 01:12:23 on dist-test-slave-21b9"
server_key: "4e050b0b6cb23ceb0714e6be956dd1cf"
server_key_iv: "da5f0c3a2501f27055b92028aaa6ff06"
server_key_version: "encryptionkey@0"
I20260812 01:12:23.926679 3889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "4ef55a1485b54be696856446dface8f9"
format_stamp: "Formatted at 2026-08-12 01:12:23 on dist-test-slave-21b9"
server_key: "4e050b0b6cb23ceb0714e6be956dd1cf"
server_key_iv: "da5f0c3a2501f27055b92028aaa6ff06"
server_key_version: "encryptionkey@0"
I20260812 01:12:23.932075 3889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:12:23.934398 3674 heartbeater.cc:503] Master 127.20.151.62:39533 was elected leader, sending a full tablet report...
I20260812 01:12:23.935726 3914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:23.937295 3889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:12:23.937451 3889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "4ef55a1485b54be696856446dface8f9"
format_stamp: "Formatted at 2026-08-12 01:12:23 on dist-test-slave-21b9"
server_key: "4e050b0b6cb23ceb0714e6be956dd1cf"
server_key_iv: "da5f0c3a2501f27055b92028aaa6ff06"
server_key_version: "encryptionkey@0"
I20260812 01:12:23.937651 3889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:23.960211 3889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:23.964376 3889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:23.964649 3889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:23.965621 3889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:23.966995 3889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:23.967070 3889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:23.967185 3889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:23.967240 3889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:23.999194 3889 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:42155
I20260812 01:12:23.999258 4105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:42155 every 8 connection(s)
I20260812 01:12:24.000556 3889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:12:24.017252 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 3889
I20260812 01:12:24.017486 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:12:24.018028 21084 external_mini_cluster.cc:1428] Setting key 642f2121469816c12d3ecc94bf47fbe5
Aug 12 01:12:24 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497143, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:12:24.026928 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:24.003351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:51291 (local address 127.20.151.62:39533)
0812 01:12:24.003853 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:24.003858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:24.004635 (+ 777us) server_negotiation.cc:407] Connection header received
0812 01:12:24.006566 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:24.006572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:24.006658 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:24.006755 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:24.008405 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:24.009251 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:24.011622 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:24.012231 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:24.012347 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:24.020534 (+ 8187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:24.020541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:24.020546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:24.020593 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:24.023724 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:24.024479 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:24.024483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:24.024485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:24.024619 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:24.025085 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:24.025088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:24.025090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:24.025408 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:24.025531 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:24.026420 (+ 889us) server_negotiation.cc:299] Negotiation successful
0812 01:12:24.026630 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":257,"threads_started":1}
I20260812 01:12:24.028043 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:24.003708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:51291)
0812 01:12:24.004467 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:24.004506 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:24.006267 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:24.007017 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:24.007025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:24.007630 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:24.008227 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:24.008247 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:24.009454 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:24.009459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:24.011420 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:24.011437 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:24.012661 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:24.012668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:24.012970 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:24.017109 (+ 4139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:24.017144 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:24.020241 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:24.023875 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:24.023883 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:24.023898 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:24.024356 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:24.024749 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:24.024753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:24.024755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:24.024973 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:24.025549 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:24.025558 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:24.025973 (+ 415us) client_negotiation.cc:769] Sending connection context
0812 01:12:24.026251 (+ 278us) client_negotiation.cc:240] Negotiation successful
0812 01:12:24.026782 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":294,"threads_started":1}
I20260812 01:12:24.029470 4107 heartbeater.cc:347] Connected to a master server at 127.20.151.62:39533
I20260812 01:12:24.029783 4107 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:24.030532 4107 heartbeater.cc:511] Master 127.20.151.62:39533 requested a full tablet report, sending...
I20260812 01:12:24.032151 3320 ts_manager.cc:194] Registered new tserver with Master: 4ef55a1485b54be696856446dface8f9 (127.20.151.3:42155)
I20260812 01:12:24.033159 3320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:51291
I20260812 01:12:24.034667 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab.
Entry for principal oryx/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab.
WARNING: no policy specified for oryx/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab.
Entry for principal oryx/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab.
WARNING: no policy specified for oryx/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab.
Entry for principal oryx/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab.
I20260812 01:12:24.230005 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 3437
Aug 12 01:12:24 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:12:24.248533 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:24.231180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.20.151.1:41571)
0812 01:12:24.231891 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:24.231898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:24.231930 (+ 32us) server_negotiation.cc:407] Connection header received
0812 01:12:24.232073 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:24.232081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:24.232319 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:24.232465 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:24.233873 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:24.235000 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:24.236903 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:24.237495 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:24.237607 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:24.242525 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:24.242539 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:24.242555 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:24.242598 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:24.245572 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:24.246302 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:24.246311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:24.246313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:24.246454 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:24.246847 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:24.246851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:24.246854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:24.247240 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:24.247597 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:24.247900 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 01:12:24.248187 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":388,"threads_started":1}
I20260812 01:12:24.249873 3668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:24.336481 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 3437
I20260812 01:12:24.366344 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 3688
Aug 12 01:12:24 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:12:24.383320 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:24.367348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.20.151.2:39511)
0812 01:12:24.367915 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:24.367923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:24.367949 (+ 26us) server_negotiation.cc:407] Connection header received
0812 01:12:24.368013 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:24.368020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:24.368233 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:24.368362 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:24.369707 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:24.370763 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:24.372312 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:24.372883 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:24.372984 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:24.377271 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:24.377286 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:24.377303 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:24.377342 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:24.380451 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:24.381191 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:24.381199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:24.381201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:24.381334 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:24.381806 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:24.381809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:24.381812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:24.382187 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:24.382433 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:24.382772 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 01:12:24.382985 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":308,"threads_started":1}
I20260812 01:12:24.384752 3871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:24.473488 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 3688
I20260812 01:12:24.506714 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 3889
Aug 12 01:12:24 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:12:24.524631 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:24.507797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.20.151.3:42155)
0812 01:12:24.508346 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:24.508353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:24.508385 (+ 32us) server_negotiation.cc:407] Connection header received
0812 01:12:24.508454 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:24.508461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:24.508665 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:24.508819 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:24.510277 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:24.511293 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:24.512526 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:24.513100 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:24.513215 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:24.517875 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:24.517889 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:24.517912 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:24.517949 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:24.521557 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:24.522230 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:24.522240 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:24.522244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:24.522414 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:24.522789 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:24.522794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:24.522798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:24.523292 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:24.523606 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:24.523930 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 01:12:24.524199 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":271,"threads_started":1}
I20260812 01:12:24.526222 4098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:24.616351 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 3889
I20260812 01:12:24.643211 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 3260
I20260812 01:12:24.644547 3378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:24.753630 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 3260
I20260812 01:12:24.778887 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:39533
--webserver_interface=127.20.151.62
--webserver_port=37969
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:39533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:25.125304 4140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:25.125656 4140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:25.125782 4140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:25.133224 4140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:12:25.133344 4140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:25.133415 4140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:25.133497 4140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:12:25.133553 4140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:12:25.143875 4140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:39533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:39533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=37969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:25.145246 4140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:25.146605 4140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:25.158329 4150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:25.158396 4151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:25.158533 4154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:25.159332 4140 server_base.cc:1034] running on GCE node
I20260812 01:12:25.160145 4140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:25.161574 4140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:25.162819 4140 hybrid_clock.cc:648] HybridClock initialized: now 1786497145162750 us; error 51 us; skew 500 ppm
I20260812 01:12:25.163201 4140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:25 dist-test-slave-21b9 krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497145, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:25.166373 4140 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:12:25.168134 4140 webserver.cc:460] Webserver started at http://127.20.151.62:37969/ using document root <none> and password file <none>
I20260812 01:12:25.168876 4140 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:25.169023 4140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:25.175123 4140 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 01:12:25.178035 4165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:25.179445 4140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:12:25.179612 4140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "16eb47f310b444c8805ae07a75547fc7"
format_stamp: "Formatted at 2026-08-12 01:12:22 on dist-test-slave-21b9"
server_key: "dc769349e420af52f38ba66d6c10422a"
server_key_iv: "bf3325b3d4198daaf968cb06633b48b3"
server_key_version: "encryptionkey@0"
I20260812 01:12:25.180603 4140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:25.198576 4140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:25.202917 4140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:25.203183 4140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:25.221813 4140 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:39533
I20260812 01:12:25.221822 4257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:39533 every 8 connection(s)
I20260812 01:12:25.223402 4140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:12:25.229660 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 4140
I20260812 01:12:25.230705 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:41571
--local_ip_for_outbound_sockets=127.20.151.1
--tserver_master_addrs=127.20.151.62:39533
--webserver_port=45201
--webserver_interface=127.20.151.1
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.151.1
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:12:25.231308 4259 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:12:25.236178 4259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7: Bootstrap starting.
I20260812 01:12:25.258505 4259 log.cc:824] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:25.267995 4259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:12:25.268536 4259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7: Bootstrap complete.
I20260812 01:12:25.277843 4259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } }
I20260812 01:12:25.278716 4259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16eb47f310b444c8805ae07a75547fc7, State: Initialized, Role: FOLLOWER
I20260812 01:12:25.279446 4259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } }
I20260812 01:12:25.279676 4259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:12:25.279847 4259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:12:25.280055 4259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 1 FOLLOWER]: Advancing to term 2
I20260812 01:12:25.284152 4259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } }
I20260812 01:12:25.284698 4259 leader_election.cc:302] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16eb47f310b444c8805ae07a75547fc7; no voters:
I20260812 01:12:25.285409 4259 leader_election.cc:288] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 01:12:25.285594 4266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 01:12:25.287750 4259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:12:25.292680 4266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [term 2 LEADER]: Becoming Leader. State: Replica: 16eb47f310b444c8805ae07a75547fc7, State: Running, Role: LEADER
I20260812 01:12:25.293413 4266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } }
I20260812 01:12:25.296267 4267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "16eb47f310b444c8805ae07a75547fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } } }
I20260812 01:12:25.296728 4267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:25.297284 4268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16eb47f310b444c8805ae07a75547fc7. Latest consensus state: current_term: 2 leader_uuid: "16eb47f310b444c8805ae07a75547fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16eb47f310b444c8805ae07a75547fc7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39533 } } }
I20260812 01:12:25.297606 4268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:25.304378 4279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:12:25.310935 4279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:12:25.312489 4279 catalog_manager.cc:1259] Loaded cluster ID: f7ca5d437e2340b0938152f4d086e0e2
I20260812 01:12:25.312676 4279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:12:25.316884 4279 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:12:25.318444 4279 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16eb47f310b444c8805ae07a75547fc7: Loaded TSK: 0
I20260812 01:12:25.320196 4279 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 01:12:25.596925 4262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:25.597219 4262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:25.597359 4262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:25.604915 4262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:25.605046 4262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:25.605196 4262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:12:25.615260 4262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.1.keytab
--principal=oryx/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:41571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=45201
--tserver_master_addrs=127.20.151.62:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:25.616675 4262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:25.618001 4262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:25.633010 4303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:25.632989 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:25.635879 4300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:25.641044 4262 server_base.cc:1034] running on GCE node
I20260812 01:12:25.641743 4262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:25.642850 4262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:25.644093 4262 hybrid_clock.cc:648] HybridClock initialized: now 1786497145644030 us; error 57 us; skew 500 ppm
I20260812 01:12:25.644423 4262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:25 dist-test-slave-21b9 krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497145, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:25.647794 4262 init.cc:377] Logged in from keytab as oryx/127.20.151.1@KRBTEST.COM (short username oryx)
I20260812 01:12:25.649420 4262 webserver.cc:460] Webserver started at http://127.20.151.1:45201/ using document root <none> and password file <none>
I20260812 01:12:25.650171 4262 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:25.650308 4262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:25.656212 4262 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 01:12:25.659219 4323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:25.660657 4262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:12:25.660832 4262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "642555a60fa345daa6d76a4489dc155f"
format_stamp: "Formatted at 2026-08-12 01:12:22 on dist-test-slave-21b9"
server_key: "2533204cde96467e7da1b6af97772068"
server_key_iv: "0d88466886ca8c12da781d03eb045521"
server_key_version: "encryptionkey@0"
I20260812 01:12:25.661842 4262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:25.690073 4262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:25.694236 4262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:25.694519 4262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:25.695601 4262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:25.697000 4262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:25.697106 4262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:25.697252 4262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:25.697342 4262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:25.729640 4262 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:41571
I20260812 01:12:25.729709 4510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:41571 every 8 connection(s)
I20260812 01:12:25.731050 4262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:12:25.734463 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 4262
I20260812 01:12:25.735262 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:39511
--local_ip_for_outbound_sockets=127.20.151.2
--tserver_master_addrs=127.20.151.62:39533
--webserver_port=44747
--webserver_interface=127.20.151.2
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.151.2
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:12:25 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:25 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 01:12:26.139659 4520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:26.140094 4520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:26.140249 4520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:26.148036 4520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:26.148180 4520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:26.148336 4520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:12:26.159049 4520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.2.keytab
--principal=oryx/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:39511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=44747
--tserver_master_addrs=127.20.151.62:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:26.160468 4520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:26.161794 4520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:26.178088 4529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:26.178439 4528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:26.178966 4532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:26.179818 4520 server_base.cc:1034] running on GCE node
I20260812 01:12:26.180498 4520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:26.181550 4520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:26.182775 4520 hybrid_clock.cc:648] HybridClock initialized: now 1786497146182734 us; error 34 us; skew 500 ppm
I20260812 01:12:26.183113 4520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:26 dist-test-slave-21b9 krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497146, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:26.186654 4520 init.cc:377] Logged in from keytab as oryx/127.20.151.2@KRBTEST.COM (short username oryx)
I20260812 01:12:26.188362 4520 webserver.cc:460] Webserver started at http://127.20.151.2:44747/ using document root <none> and password file <none>
I20260812 01:12:26.189141 4520 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:26.189299 4520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:26.194974 4520 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 01:12:26.198202 4543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:26.199833 4520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 01:12:26.200013 4520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "55b812d1684243f69e774dfbd8d273ea"
format_stamp: "Formatted at 2026-08-12 01:12:23 on dist-test-slave-21b9"
server_key: "704a3f04ed74dc083f1426e333fd760e"
server_key_iv: "11c7d2fd8b7ea4122e0803ea1b1a4497"
server_key_version: "encryptionkey@0"
I20260812 01:12:26.201001 4520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:26.228686 4520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:26.233168 4520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:26.233451 4520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:26.234593 4520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:26.236132 4520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:26.236246 4520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:26.236354 4520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:26.236426 4520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:26.272037 4520 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:39511
I20260812 01:12:26.272095 4737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:39511 every 8 connection(s)
I20260812 01:12:26.273524 4520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:12:26.278093 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 4520
I20260812 01:12:26.278658 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:42155
--local_ip_for_outbound_sockets=127.20.151.3
--tserver_master_addrs=127.20.151.62:39533
--webserver_port=46097
--webserver_interface=127.20.151.3
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.151.3
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:12:26 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:26 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 01:12:26.632967 4743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:26.633267 4743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:26.633409 4743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:26.640772 4743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:26.640928 4743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:26.641094 4743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:12:26.651132 4743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/krb5kdc/oryx_127.20.151.3.keytab
--principal=oryx/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:42155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=46097
--tserver_master_addrs=127.20.151.62:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:26.652585 4743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:26.653931 4743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:26.669164 4755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:26.671739 4754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:26.681391 4757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:26.681375 4743 server_base.cc:1034] running on GCE node
I20260812 01:12:26.682262 4743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:26.683393 4743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:26.684619 4743 hybrid_clock.cc:648] HybridClock initialized: now 1786497146684557 us; error 61 us; skew 500 ppm
I20260812 01:12:26.684963 4743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:26 dist-test-slave-21b9 krb5kdc[3237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497146, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:26.688375 4743 init.cc:377] Logged in from keytab as oryx/127.20.151.3@KRBTEST.COM (short username oryx)
I20260812 01:12:26.692615 4743 webserver.cc:460] Webserver started at http://127.20.151.3:46097/ using document root <none> and password file <none>
I20260812 01:12:26.693359 4743 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:26.693497 4743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:26.699307 4743 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 01:12:26.702091 4771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:26.703504 4743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 01:12:26.703675 4743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "4ef55a1485b54be696856446dface8f9"
format_stamp: "Formatted at 2026-08-12 01:12:23 on dist-test-slave-21b9"
server_key: "4e050b0b6cb23ceb0714e6be956dd1cf"
server_key_iv: "da5f0c3a2501f27055b92028aaa6ff06"
server_key_version: "encryptionkey@0"
I20260812 01:12:26.704736 4743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:26.754341 4743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:26.758852 4743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:26.759120 4743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:26.760245 4743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:26.761767 4743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:26.761870 4743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:26.761986 4743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:26.762064 4743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:26.802456 4743 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:42155
I20260812 01:12:26.802517 4977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:42155 every 8 connection(s)
I20260812 01:12:26.804117 4743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:12:26.804535 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 4743
Aug 12 01:12:26 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:26 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:26.845021 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:26.806317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.20.151.62:39533)
0812 01:12:26.815052 (+ 8735us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:26.815058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:26.821366 (+ 6308us) server_negotiation.cc:407] Connection header received
0812 01:12:26.821408 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:26.821413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:26.821471 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:26.821575 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:26.824156 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.825649 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.827866 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.828660 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.828785 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:26.833133 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:26.833149 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:26.833177 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:26.833231 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:26.837868 (+ 4637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:26.839053 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:26.839065 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:26.839078 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:26.839424 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:26.839974 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:26.839981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:26.839985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:26.840824 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:26.841203 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:26.843030 (+ 1827us) server_negotiation.cc:299] Negotiation successful
0812 01:12:26.843704 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8593,"thread_start_us":14093,"threads_started":1}
W20260812 01:12:26.947135 4519 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.201s user 0.000s sys 0.015s
W20260812 01:12:26.949116 4518 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:25.734074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:55157 (local address 127.20.151.62:39533)
0812 01:12:25.734769 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:25.734780 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:25.735466 (+ 686us) server_negotiation.cc:407] Connection header received
0812 01:12:25.738709 (+ 3243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:25.738734 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:25.739113 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:25.739483 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:25.741436 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:25.742739 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:25.744387 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:25.745075 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:25.745248 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:26.948823 (+1203575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:55157: BlockingRecv error: recv got EOF from 127.20.151.1:55157 (error 108)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":254,"threads_started":1}
I20260812 01:12:26.949790 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:25.734156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:55157)
0812 01:12:25.735248 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:25.735315 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:25.737980 (+ 2665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:25.740189 (+ 2209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:25.740196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:25.740705 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:25.741239 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:25.741258 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:25.742915 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:25.742921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:25.744232 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:25.744246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:25.745308 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:25.745317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:25.745578 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:25.746223 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:25.746246 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:26.948164 (+1201918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":669,"threads_started":1}
W20260812 01:12:26.951020 4514 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:39533 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.151.62:39533: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Aug 12 01:12:26 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:26 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 01:12:26.963647 4518 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:26.952075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:59623 (local address 127.20.151.62:39533)
0812 01:12:26.952246 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:26.952252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:26.952418 (+ 166us) server_negotiation.cc:407] Connection header received
0812 01:12:26.952696 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:26.952702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:26.952787 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:26.952857 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:26.954100 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.955333 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.957411 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.958088 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.958185 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:26.963363 (+ 5178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:59623: BlockingRecv error: recv got EOF from 127.20.151.1:59623 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:12:26.964227 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:26.951769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:59623)
0812 01:12:26.952327 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:26.952348 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:26.952546 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:26.953096 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:26.953100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:26.953662 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:26.953905 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:26.953918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.955580 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.955586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:26.957228 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:26.957246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.958502 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.958507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:26.958673 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:26.959380 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:26.959402 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:26.963985 (+ 4583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":7488,"threads_started":1}
Aug 12 01:12:26 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:26 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:26.973599 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:26.965220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:49919)
0812 01:12:26.965356 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:26.965404 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:26.965520 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:26.965822 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:26.965827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:26.966182 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:26.966348 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:26.966357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.967371 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.967376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:26.968368 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:26.968381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.969109 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.969115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:26.969287 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:26.969846 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:26.969863 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:26.973319 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 01:12:26.973764 4518 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:26.965028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:49919 (local address 127.20.151.62:39533)
0812 01:12:26.965174 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:26.965178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:26.965473 (+ 295us) server_negotiation.cc:407] Connection header received
0812 01:12:26.965624 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:26.965630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:26.965706 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:26.965785 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:26.966508 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.967222 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.968505 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.968965 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.969080 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:26.973524 (+ 4444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:49919: BlockingRecv error: recv got EOF from 127.20.151.1:49919 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260812 01:12:26.974212 4514 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 01:12:27.521572 4742 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.231s user 0.004s sys 0.019s
W20260812 01:12:27.523075 4741 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:26.276313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:39485 (local address 127.20.151.62:39533)
0812 01:12:26.276899 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:26.276907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:26.277913 (+ 1006us) server_negotiation.cc:407] Connection header received
0812 01:12:26.281179 (+ 3266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:26.281185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:26.281287 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:26.281422 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:26.283961 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.285040 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.287796 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.288364 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.288506 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.522718 (+1234212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:39485: BlockingRecv error: recv got EOF from 127.20.151.2:39485 (error 108)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":258,"threads_started":1}
I20260812 01:12:27.523696 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:26.276767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:39485)
0812 01:12:26.277687 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:26.277767 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:26.280766 (+ 2999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:26.281730 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:26.281745 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:26.282620 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:26.283729 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:26.283760 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.285228 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.285235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:26.287605 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:26.287627 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.288650 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.288659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:26.289064 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:26.290054 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:26.290094 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:27.522410 (+1232316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":313,"threads_started":1}
W20260812 01:12:27.524715 4739 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:39533 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.151.62:39533: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Aug 12 01:12:27 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:27 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:27.536507 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:27.525333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:52671)
0812 01:12:27.525579 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:27.525606 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:27.525762 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:27.526195 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:27.526200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:27.526763 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:27.527033 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:27.527048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.528131 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.528137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:27.529875 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:27.529892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.530610 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.530616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:27.530812 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.531543 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:27.531570 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:27.536214 (+ 4644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 01:12:27.536755 4741 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:27.525454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:52671 (local address 127.20.151.62:39533)
0812 01:12:27.525597 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:27.525602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:27.525677 (+ 75us) server_negotiation.cc:407] Connection header received
0812 01:12:27.525930 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:27.525935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:27.526048 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:27.526153 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:27.527200 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.527997 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.530041 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.530431 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.530533 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.536500 (+ 5967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:52671: BlockingRecv error: recv got EOF from 127.20.151.2:52671 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 12 01:12:27 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:27 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 01:12:27.547698 4741 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:27.537705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:45057 (local address 127.20.151.62:39533)
0812 01:12:27.537903 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:27.537909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:27.537930 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:12:27.537990 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:27.537996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:27.538093 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:27.538209 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:27.538936 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.540233 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.541480 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.542199 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.542331 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.547308 (+ 4977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:45057: BlockingRecv error: recv got EOF from 127.20.151.2:45057 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:12:27.547989 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:27.537428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:45057)
0812 01:12:27.537590 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:27.537608 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:27.537691 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:27.538312 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:27.538316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:27.538639 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:27.538803 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:27.538813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.540389 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.540393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:27.541358 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:27.541370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.542344 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.542349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:27.542490 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.543173 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:27.543190 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:27.547726 (+ 4536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 01:12:27.548341 4739 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 01:12:27.900163 4986 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.070s user 0.006s sys 0.012s
W20260812 01:12:27.901449 4983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:26.809896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:36993 (local address 127.20.151.62:39533)
0812 01:12:26.820760 (+ 10864us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:26.820767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:26.820794 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:12:26.820875 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:26.820882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:26.820974 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:26.821110 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:26.823548 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.824823 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.827461 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.828169 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.828284 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.901196 (+1072912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:36993: BlockingRecv error: recv got EOF from 127.20.151.3:36993 (error 108)
Metrics: {"server-negotiator.queue_time_us":10699,"spinlock_wait_cycles":11648,"thread_start_us":4911,"threads_started":1}
I20260812 01:12:27.901897 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:26.809116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:36993)
0812 01:12:26.810100 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:26.810160 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:26.812771 (+ 2611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:26.821481 (+ 8710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:26.821495 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:26.822350 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:26.823308 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:26.823342 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.825040 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.825046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:26.827226 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:26.827258 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:26.829234 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:26.829241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:26.829554 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:26.830338 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:26.830368 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:27.900945 (+1070577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":722,"threads_started":1}
W20260812 01:12:27.902874 4979 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:39533 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.151.62:39533: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Aug 12 01:12:27 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:27 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:27.912587 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:27.903601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:35661)
0812 01:12:27.903813 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:27.903837 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:27.903976 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:27.904333 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:27.904338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:27.904745 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:27.904915 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:27.904925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.906047 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.906052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:27.907027 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:27.907039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.907846 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.907853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:27.908007 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.908626 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:27.908644 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:27.912274 (+ 3630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 01:12:27.912734 4983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:27.903652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:35661 (local address 127.20.151.62:39533)
0812 01:12:27.903850 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:27.903855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:27.903914 (+ 59us) server_negotiation.cc:407] Connection header received
0812 01:12:27.904104 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:27.904109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:27.904190 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:27.904281 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:27.905075 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.905905 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.907168 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.907675 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.907788 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.912509 (+ 4721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:35661: BlockingRecv error: recv got EOF from 127.20.151.3:35661 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 01:12:27 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:27 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:27.927170 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:27.913277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:50821)
0812 01:12:27.913412 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:27.913427 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:27.913507 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:27.913951 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:27.913956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:27.914256 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:27.914410 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:27.914418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.915445 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.915449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:27.916455 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:27.916467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.917194 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.917200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:27.917358 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.923304 (+ 5946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:27.923332 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:27.926941 (+ 3609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 01:12:27.927493 4983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:27.913397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:50821 (local address 127.20.151.62:39533)
0812 01:12:27.913679 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:27.913684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:27.913700 (+ 16us) server_negotiation.cc:407] Connection header received
0812 01:12:27.913744 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:27.913748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:27.913816 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:27.913895 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:27.914544 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.915301 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.916593 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.917035 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.917150 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.927186 (+ 10036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:50821: BlockingRecv error: recv got EOF from 127.20.151.3:50821 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
W20260812 01:12:27.927698 4979 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 01:12:27 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:27 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:27.983811 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:27.974824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:53945)
0812 01:12:27.975415 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:27.975431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:27.975571 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:27.975926 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:27.975930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:27.976374 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:27.976523 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:27.976532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.977529 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.977534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:27.978470 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:27.978482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.979203 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.979209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:27.979402 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.979992 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:27.980010 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:27.983426 (+ 3416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":331,"threads_started":1}
W20260812 01:12:27.983877 4983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:27.974889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:53945 (local address 127.20.151.62:39533)
0812 01:12:27.975153 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:27.975160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:27.975517 (+ 357us) server_negotiation.cc:407] Connection header received
0812 01:12:27.975733 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:27.975737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:27.975800 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:27.975880 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:27.976683 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.977386 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.978609 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:27.979024 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:27.979140 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:27.983663 (+ 4523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:53945: BlockingRecv error: recv got EOF from 127.20.151.1:53945 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 12 01:12:28 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:28 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:28.559087 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:28.549037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:33001)
0812 01:12:28.549700 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:28.549719 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:28.549845 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:28.550356 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:28.550360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:28.550823 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:28.550985 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:28.550996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.552506 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.552511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:28.553829 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:28.553848 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.554633 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.554638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:28.554762 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:28.555375 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:28.555392 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:28.558839 (+ 3447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":357,"threads_started":1}
W20260812 01:12:28.559409 4996 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:28.549227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:33001 (local address 127.20.151.62:39533)
0812 01:12:28.550029 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:28.550034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:28.550054 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:12:28.550113 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:28.550117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:28.550180 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:28.550280 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:28.551167 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.552286 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.554011 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.554444 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.554549 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:28.559090 (+ 4541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:33001: BlockingRecv error: recv got EOF from 127.20.151.2:33001 (error 108)
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":576,"threads_started":1}
Aug 12 01:12:28 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:28 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:28.937693 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:28.928345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:32823)
0812 01:12:28.928929 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:28.928946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:28.929085 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:28.929438 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:28.929442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:28.929893 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:28.930057 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:28.930068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.931229 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.931233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:28.932226 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:28.932239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.932975 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.932981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:28.933147 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:28.933745 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:28.933763 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:28.937434 (+ 3671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":343,"threads_started":1}
W20260812 01:12:28.937845 4996 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:28.928530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:32823 (local address 127.20.151.62:39533)
0812 01:12:28.928759 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:28.928765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:28.929045 (+ 280us) server_negotiation.cc:407] Connection header received
0812 01:12:28.929225 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:28.929231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:28.929299 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:28.929381 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:28.930210 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.931093 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.932373 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.932793 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.932905 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:28.937641 (+ 4736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:32823: BlockingRecv error: recv got EOF from 127.20.151.3:32823 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 12 01:12:28 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:28 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:28.995021 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:28.985147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:34667)
0812 01:12:28.985908 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:28.985926 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:28.986070 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:28.986475 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:28.986481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:28.987078 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:28.987369 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:28.987385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.988601 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.988605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:28.989674 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:28.989688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.990421 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.990425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:28.990550 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:28.991237 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:28.991255 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:28.994760 (+ 3505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":496,"threads_started":1}
W20260812 01:12:28.995338 4996 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:28.985210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:34667 (local address 127.20.151.62:39533)
0812 01:12:28.985476 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:28.985485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:28.986022 (+ 537us) server_negotiation.cc:407] Connection header received
0812 01:12:28.986208 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:28.986215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:28.986309 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:28.986424 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:28.987525 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.988433 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.989842 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:28.990279 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:28.990380 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:28.994981 (+ 4601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:34667: BlockingRecv error: recv got EOF from 127.20.151.1:34667 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 12 01:12:29 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:29 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:29.569725 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:29.560301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:59737)
0812 01:12:29.560916 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:29.560941 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:29.561090 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:29.561491 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:29.561495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:29.561948 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:29.562179 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:29.562190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:29.563420 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:29.563424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:29.564347 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:29.564359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:29.565152 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:29.565157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:29.565295 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:29.565836 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:29.565856 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:29.569463 (+ 3607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":317,"threads_started":1}
W20260812 01:12:29.570050 5003 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:29.560419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:59737 (local address 127.20.151.62:39533)
0812 01:12:29.561164 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:29.561170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:29.561189 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:12:29.561247 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:29.561252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:29.561326 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:29.561418 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:29.562339 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:29.563315 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:29.564506 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:29.565013 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:29.565116 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:29.569774 (+ 4658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:59737: BlockingRecv error: recv got EOF from 127.20.151.2:59737 (error 108)
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":514,"threads_started":1}
Aug 12 01:12:29 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:29 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:29.948191 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:29.938666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:52373)
0812 01:12:29.939189 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:29.939206 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:29.939367 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:29.939770 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:29.939773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:29.940210 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:29.940401 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:29.940411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:29.941555 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:29.941559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:29.942534 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:29.942548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:29.943391 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:29.943396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:29.943536 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:29.944224 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:29.944249 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:29.947919 (+ 3670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":283,"threads_started":1}
W20260812 01:12:29.948367 5003 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:29.938885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:52373 (local address 127.20.151.62:39533)
0812 01:12:29.939093 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:29.939099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:29.939296 (+ 197us) server_negotiation.cc:407] Connection header received
0812 01:12:29.939555 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:29.939559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:29.939634 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:29.939736 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:29.940628 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:29.941387 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:29.942672 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:29.943211 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:29.943368 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:29.948140 (+ 4772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:52373: BlockingRecv error: recv got EOF from 127.20.151.3:52373 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 12 01:12:29 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:29 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:30.005667 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:29.996096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:39703)
0812 01:12:29.996598 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:29.996616 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:29.996734 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:29.997055 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:29.997058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:29.997510 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:29.997723 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:29.997734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:29.998774 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:29.998780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:29.999779 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:29.999793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:30.000565 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:30.000569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:30.000709 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:30.001311 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:30.001329 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:30.005173 (+ 3844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":241,"threads_started":1}
W20260812 01:12:30.005708 5003 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:29.996173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:39703 (local address 127.20.151.62:39533)
0812 01:12:29.996377 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:29.996383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:29.996688 (+ 305us) server_negotiation.cc:407] Connection header received
0812 01:12:29.996823 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:29.996827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:29.996904 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:29.996983 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:29.997871 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:29.998643 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:29.999936 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:30.000383 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:30.000534 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:30.005432 (+ 4898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:39703: BlockingRecv error: recv got EOF from 127.20.151.1:39703 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 01:12:30 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:30 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:30.580204 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:30.570755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:38347)
0812 01:12:30.571345 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:30.571363 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:30.571475 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:30.571877 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:30.571881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:30.572326 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:30.572596 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:30.572612 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:30.573858 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:30.573862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:30.574826 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:30.574839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:30.575535 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:30.575539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:30.575700 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:30.576281 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:30.576298 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:30.579836 (+ 3538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":301,"threads_started":1}
W20260812 01:12:30.580375 5012 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:30.570872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:38347 (local address 127.20.151.62:39533)
0812 01:12:30.571405 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:30.571409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:30.571428 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:12:30.571591 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:30.571596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:30.571722 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:30.571850 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:30.572745 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:30.573728 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:30.574939 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:30.575407 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:30.575502 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:30.580105 (+ 4603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:38347: BlockingRecv error: recv got EOF from 127.20.151.2:38347 (error 108)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":270,"threads_started":1}
Aug 12 01:12:30 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:30 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:30.958184 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:30.949294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:51857)
0812 01:12:30.949845 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:30.949863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:30.949994 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:30.950306 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:30.950309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:30.950696 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:30.950879 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:30.950888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:30.951936 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:30.951939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:30.952855 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:30.952867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:30.953460 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:30.953463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:30.953594 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:30.954154 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:30.954170 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:30.957922 (+ 3752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":319,"threads_started":1}
W20260812 01:12:30.958276 5012 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:30.949444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:51857 (local address 127.20.151.62:39533)
0812 01:12:30.949619 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:30.949624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:30.949942 (+ 318us) server_negotiation.cc:407] Connection header received
0812 01:12:30.950103 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:30.950106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:30.950175 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:30.950262 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:30.951016 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:30.951819 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:30.952964 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:30.953354 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:30.953442 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:30.958088 (+ 4646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:51857: BlockingRecv error: recv got EOF from 127.20.151.3:51857 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 01:12:31 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:31 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:31.015785 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:31.006748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:36047)
0812 01:12:31.007407 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:31.007423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:31.007537 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:31.007837 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:31.007840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:31.008249 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:31.008464 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:31.008477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.009543 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.009548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:31.010594 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:31.010611 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.011333 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.011338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:31.011470 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:31.012059 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:31.012082 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:31.015519 (+ 3437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":369,"threads_started":1}
W20260812 01:12:31.015980 5012 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:31.006855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:36047 (local address 127.20.151.62:39533)
0812 01:12:31.007122 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:31.007128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:31.007535 (+ 407us) server_negotiation.cc:407] Connection header received
0812 01:12:31.007638 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:31.007641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:31.007719 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:31.007795 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:31.008630 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.009404 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.010747 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.011175 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.011303 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:31.015739 (+ 4436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:36047: BlockingRecv error: recv got EOF from 127.20.151.1:36047 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Aug 12 01:12:31 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:31 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:31.590745 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:31.581380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:51599)
0812 01:12:31.582035 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:31.582058 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:31.582210 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:31.582619 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:31.582623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:31.583031 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:31.583208 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:31.583217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.584292 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.584296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:31.585320 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:31.585333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.586036 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.586041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:31.586213 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:31.586803 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:31.586828 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:31.590360 (+ 3532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":360,"threads_started":1}
W20260812 01:12:31.590842 5021 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:31.581483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:51599 (local address 127.20.151.62:39533)
0812 01:12:31.582319 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:31.582325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:31.582343 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:12:31.582392 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:31.582396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:31.582455 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:31.582547 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:31.583377 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.584154 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.585461 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.585891 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.585997 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:31.590599 (+ 4602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:51599: BlockingRecv error: recv got EOF from 127.20.151.2:51599 (error 108)
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":569,"threads_started":1}
Aug 12 01:12:31 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:31 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:31.968995 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:31.959176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:39551)
0812 01:12:31.959829 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:31.959849 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:31.959967 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:31.960288 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:31.960290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:31.960690 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:31.960855 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:31.960863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.961946 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.961950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:31.962990 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:31.963003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.963721 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.963726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:31.963861 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:31.964494 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:31.964526 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:31.968607 (+ 4081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":413,"threads_started":1}
W20260812 01:12:31.969048 5021 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:31.959332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:39551 (local address 127.20.151.62:39533)
0812 01:12:31.959552 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:31.959557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:31.959940 (+ 383us) server_negotiation.cc:407] Connection header received
0812 01:12:31.960070 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:31.960074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:31.960142 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:31.960237 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:31.961006 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.961787 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.963141 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:31.963581 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:31.963678 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:31.968825 (+ 5147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:39551: BlockingRecv error: recv got EOF from 127.20.151.3:39551 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 12 01:12:32 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:32 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:32.026121 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:32.016835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:60039)
0812 01:12:32.017488 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:32.017511 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:32.017661 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:32.018046 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:32.018050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:32.018636 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:32.018850 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:32.018866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.019914 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.019918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:32.020855 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:32.020867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.021518 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.021522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:32.021673 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:32.022240 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:32.022256 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:32.025762 (+ 3506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"spinlock_wait_cycles":32000,"thread_start_us":404,"threads_started":1}
W20260812 01:12:32.026289 5021 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:32.017001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:60039 (local address 127.20.151.62:39533)
0812 01:12:32.017185 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:32.017190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:32.017596 (+ 406us) server_negotiation.cc:407] Connection header received
0812 01:12:32.017801 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:32.017807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:32.017896 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:32.017974 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:32.018999 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.019805 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.021001 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.021409 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.021532 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:32.026011 (+ 4479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:60039: BlockingRecv error: recv got EOF from 127.20.151.1:60039 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 12 01:12:32 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:32 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:32.600412 5025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:32.591768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:60731)
0812 01:12:32.592326 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:32.592343 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:32.592460 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:32.592771 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:32.592774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:32.593214 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:32.593367 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:32.593375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.594348 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.594351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:32.595386 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:32.595400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.596061 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.596065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:32.596239 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:32.596865 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:32.596885 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:32.600158 (+ 3273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":257,"threads_started":1}
W20260812 01:12:32.600610 5027 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:32.591967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:60731 (local address 127.20.151.62:39533)
0812 01:12:32.592424 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:32.592429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:32.592446 (+ 17us) server_negotiation.cc:407] Connection header received
0812 01:12:32.592553 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:32.592557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:32.592647 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:32.592735 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:32.593498 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.594243 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.595520 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.595943 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.596042 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:32.600376 (+ 4334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:60731: BlockingRecv error: recv got EOF from 127.20.151.2:60731 (error 108)
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":247,"threads_started":1}
Aug 12 01:12:32 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:32 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:32.982366 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:32.970090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:53325)
0812 01:12:32.970652 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:32.970674 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:32.970832 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:32.971210 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:32.971216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:32.971721 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:32.971877 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:32.971887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.972944 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.972948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:32.973895 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:32.973907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.974599 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.974605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:32.974786 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:32.978525 (+ 3739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:32.978545 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:32.982039 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":266,"threads_started":1}
W20260812 01:12:32.982604 5027 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:32.970223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:53325 (local address 127.20.151.62:39533)
0812 01:12:32.970431 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:32.970438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:32.970778 (+ 340us) server_negotiation.cc:407] Connection header received
0812 01:12:32.970955 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:32.970960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:32.971047 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:32.971147 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:32.972051 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.972797 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.974036 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:32.974446 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:32.974567 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:32.982323 (+ 7756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:53325: BlockingRecv error: recv got EOF from 127.20.151.3:53325 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 12 01:12:33 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:33 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:33.037783 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:33.027338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:50283)
0812 01:12:33.028471 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:33.028492 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:33.028635 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:33.029046 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:33.029051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:33.029513 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:33.029703 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:33.029714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.030850 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.030855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:33.031936 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:33.031953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.032870 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.032876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:33.033049 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:33.033681 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:33.033699 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:33.037387 (+ 3688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":805,"threads_started":1}
W20260812 01:12:33.037914 5027 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:33.027367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:50283 (local address 127.20.151.62:39533)
0812 01:12:33.027586 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:33.027592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:33.028589 (+ 997us) server_negotiation.cc:407] Connection header received
0812 01:12:33.028790 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:33.028794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:33.028863 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:33.028951 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:33.029862 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.030703 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.032114 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.032690 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.032809 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:33.037652 (+ 4843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:50283: BlockingRecv error: recv got EOF from 127.20.151.1:50283 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 12 01:12:33 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:33 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:33.611116 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:33.601420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:39185)
0812 01:12:33.601996 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:33.602016 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:33.602136 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:33.602414 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:33.602416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:33.602855 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:33.603030 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:33.603040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.604160 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.604164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:33.605225 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:33.605241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.606079 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.606084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:33.606231 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:33.606845 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:33.606864 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:33.610841 (+ 3977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":341,"threads_started":1}
W20260812 01:12:33.611513 5032 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:33.601462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:39185 (local address 127.20.151.62:39533)
0812 01:12:33.601945 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:33.601950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:33.602081 (+ 131us) server_negotiation.cc:407] Connection header received
0812 01:12:33.602229 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:33.602232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:33.602292 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:33.602382 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:33.603194 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.603996 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.605369 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.605935 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.606039 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:33.611172 (+ 5133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:39185: BlockingRecv error: recv got EOF from 127.20.151.2:39185 (error 108)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":286,"threads_started":1}
Aug 12 01:12:33 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:33 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:33.994560 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:33.983698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:44189)
0812 01:12:33.984210 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:33.984226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:33.984370 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:33.984768 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:33.984773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:33.985310 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:33.985541 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:33.985553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.986908 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.986915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:33.988495 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:33.988515 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.989309 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.989314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:33.989501 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:33.990148 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:33.990174 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:33.994231 (+ 4057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":307,"threads_started":1}
W20260812 01:12:33.994724 5032 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:33.983758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:44189 (local address 127.20.151.62:39533)
0812 01:12:33.983924 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:33.983930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:33.984317 (+ 387us) server_negotiation.cc:407] Connection header received
0812 01:12:33.984523 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:33.984527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:33.984592 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:33.984676 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:33.985695 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.986736 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.988672 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:33.989140 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:33.989283 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:33.994505 (+ 5222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:44189: BlockingRecv error: recv got EOF from 127.20.151.3:44189 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 12 01:12:34 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:34 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:34.048712 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:34.038902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:47417)
0812 01:12:34.039558 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:34.039609 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:34.039750 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:34.040114 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:34.040118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:34.040581 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:34.040770 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:34.040781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:34.041886 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:34.041892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:34.042859 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:34.042872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:34.043660 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:34.043666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:34.043826 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:34.044428 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:34.044446 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:34.048353 (+ 3907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":307,"threads_started":1}
W20260812 01:12:34.048933 5032 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:34.038966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:47417 (local address 127.20.151.62:39533)
0812 01:12:34.039148 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:34.039152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:34.039700 (+ 548us) server_negotiation.cc:407] Connection header received
0812 01:12:34.039870 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:34.039876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:34.039960 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:34.040081 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:34.040931 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:34.041730 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:34.043004 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:34.043488 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:34.043595 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:34.048636 (+ 5041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:47417: BlockingRecv error: recv got EOF from 127.20.151.1:47417 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 12 01:12:34 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:34 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:34.621806 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:34.612197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:32995)
0812 01:12:34.613285 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:34.613301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:34.613421 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:34.613851 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:34.613854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:34.614331 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:34.614526 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:34.614537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:34.615629 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:34.615633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:34.616590 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:34.616603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:34.617314 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:34.617320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:34.617482 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:34.618074 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:34.618091 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:34.621454 (+ 3363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":985,"thread_start_us":734,"threads_started":1}
W20260812 01:12:34.622022 5035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:34.612317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:32995 (local address 127.20.151.62:39533)
0812 01:12:34.613082 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:34.613088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:34.613372 (+ 284us) server_negotiation.cc:407] Connection header received
0812 01:12:34.613581 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:34.613585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:34.613678 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:34.613793 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:34.614678 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:34.615489 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:34.616732 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:34.617161 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:34.617272 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:34.621707 (+ 4435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:32995: BlockingRecv error: recv got EOF from 127.20.151.2:32995 (error 108)
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":511,"threads_started":1}
Aug 12 01:12:35 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:35 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:35.005811 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:34.995998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:44765)
0812 01:12:34.996829 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:34.996847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:34.996993 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:34.997345 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:34.997348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:34.997907 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:34.998098 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:34.998109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:34.999179 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:34.999183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:35.000315 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:35.000329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:35.001098 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:35.001103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:35.001233 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:35.001889 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:35.001908 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:35.005519 (+ 3611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":413,"threads_started":1}
W20260812 01:12:35.006069 5035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:34.996264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:44765 (local address 127.20.151.62:39533)
0812 01:12:34.996485 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:34.996492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:34.996924 (+ 432us) server_negotiation.cc:407] Connection header received
0812 01:12:34.997117 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:34.997121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:34.997187 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:34.997284 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:34.998237 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:34.999026 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:35.000471 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:35.000912 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:35.001028 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:35.005771 (+ 4743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:44765: BlockingRecv error: recv got EOF from 127.20.151.3:44765 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 12 01:12:35 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:35 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:35.059379 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:35.049980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:43959)
0812 01:12:35.050604 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:35.050622 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:35.050764 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:35.051111 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:35.051116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:35.051583 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:35.051766 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:35.051776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:35.052821 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:35.052825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:35.053770 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:35.053782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:35.054516 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:35.054523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:35.054676 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:35.055380 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:35.055400 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:35.058987 (+ 3587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":309,"threads_started":1}
W20260812 01:12:35.059794 5035 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:35.050202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:43959 (local address 127.20.151.62:39533)
0812 01:12:35.050516 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:35.050522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:35.050708 (+ 186us) server_negotiation.cc:407] Connection header received
0812 01:12:35.050885 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:35.050890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:35.050967 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:35.051054 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:35.051919 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:35.052676 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:35.053917 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:35.054332 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:35.054454 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:35.059265 (+ 4811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:43959: BlockingRecv error: recv got EOF from 127.20.151.1:43959 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 12 01:12:35 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:35 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:35.632647 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:35.622903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:37419)
0812 01:12:35.623574 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:35.623597 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:35.623744 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:35.624183 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:35.624187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:35.624735 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:35.624965 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:35.624978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:35.626284 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:35.626288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:35.627263 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:35.627319 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:35.628090 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:35.628097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:35.628260 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:35.628853 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:35.628870 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:35.632340 (+ 3470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":307,"threads_started":1}
W20260812 01:12:35.632884 5044 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:35.623065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:37419 (local address 127.20.151.62:39533)
0812 01:12:35.623818 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:35.623824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:35.623841 (+ 17us) server_negotiation.cc:407] Connection header received
0812 01:12:35.623908 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:35.623913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:35.624018 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:35.624128 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:35.625120 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:35.626146 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:35.627455 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:35.627888 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:35.628002 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:35.632600 (+ 4598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:37419: BlockingRecv error: recv got EOF from 127.20.151.2:37419 (error 108)
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":518,"threads_started":1}
Aug 12 01:12:36 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:36 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:36.017570 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:36.006809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:57199)
0812 01:12:36.007343 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:36.007361 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:36.007481 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:36.007757 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:36.007760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:36.008166 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:36.008387 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:36.008401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.009960 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.009966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:36.011499 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:36.011516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.012415 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.012422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:36.012601 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:36.013307 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:36.013331 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:36.017182 (+ 3851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":251,"threads_started":1}
W20260812 01:12:36.017644 5044 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:36.006877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:57199 (local address 127.20.151.62:39533)
0812 01:12:36.007101 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:36.007107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:36.007431 (+ 324us) server_negotiation.cc:407] Connection header received
0812 01:12:36.007576 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:36.007580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:36.007646 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:36.007725 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:36.008570 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.009813 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.011671 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.012210 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.012337 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:36.017408 (+ 5071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:57199: BlockingRecv error: recv got EOF from 127.20.151.3:57199 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 01:12:36 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:36 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:36.070091 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:36.060477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:58735)
0812 01:12:36.061022 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:36.061037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:36.061176 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:36.061505 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:36.061510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:36.061994 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:36.062150 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:36.062160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.063252 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.063256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:36.064452 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:36.064468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.065366 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.065372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:36.065539 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:36.066155 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:36.066174 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:36.069838 (+ 3664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":287,"threads_started":1}
W20260812 01:12:36.070363 5044 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:36.060700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:58735 (local address 127.20.151.62:39533)
0812 01:12:36.060942 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:36.060949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:36.061114 (+ 165us) server_negotiation.cc:407] Connection header received
0812 01:12:36.061295 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:36.061300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:36.061365 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:36.061451 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:36.062303 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.063102 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.064609 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.065207 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.065312 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:36.070102 (+ 4790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:58735: BlockingRecv error: recv got EOF from 127.20.151.1:58735 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 01:12:36 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:36 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:36.643091 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:36.633658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:39519)
0812 01:12:36.634228 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:36.634247 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:36.634363 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:36.634679 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:36.634682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:36.635103 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:36.635296 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:36.635312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.636297 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.636301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:36.637302 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:36.637316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.638075 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.638080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:36.638216 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:36.638889 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:36.638907 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:36.642767 (+ 3860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":312,"threads_started":1}
W20260812 01:12:36.643332 5053 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:36.633721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:39519 (local address 127.20.151.62:39533)
0812 01:12:36.634245 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:36.634249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:36.634314 (+ 65us) server_negotiation.cc:407] Connection header received
0812 01:12:36.634472 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:36.634475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:36.634540 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:36.634631 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:36.635419 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.636138 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.637438 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:36.637914 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:36.638005 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:36.643015 (+ 5010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:39519: BlockingRecv error: recv got EOF from 127.20.151.2:39519 (error 108)
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":241,"threads_started":1}
Aug 12 01:12:37 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:37 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:37.027668 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:37.018707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:53221)
0812 01:12:37.019503 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:37.019524 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:37.019659 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:37.019995 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:37.020000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:37.020452 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:37.020613 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:37.020623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.021634 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.021640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:37.022612 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:37.022624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.023325 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.023332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:37.023511 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:37.024074 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:37.024091 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:37.027429 (+ 3338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":476,"threads_started":1}
W20260812 01:12:37.027841 5053 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:37.018788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:53221 (local address 127.20.151.62:39533)
0812 01:12:37.019004 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:37.019011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:37.019610 (+ 599us) server_negotiation.cc:407] Connection header received
0812 01:12:37.019787 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:37.019793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:37.019858 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:37.019940 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:37.020764 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.021494 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.022756 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.023145 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.023252 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:37.027620 (+ 4368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:53221: BlockingRecv error: recv got EOF from 127.20.151.3:53221 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 12 01:12:37 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:37 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:37.080097 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:37.071006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:44347)
0812 01:12:37.071529 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:37.071545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:37.071679 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:37.072025 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:37.072030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:37.072486 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:37.072657 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:37.072667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.073689 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.073695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:37.074683 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:37.074696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.075427 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.075434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:37.075600 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:37.076210 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:37.076228 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:37.079776 (+ 3548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":270,"threads_started":1}
W20260812 01:12:37.080214 5053 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:37.071181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:44347 (local address 127.20.151.62:39533)
0812 01:12:37.071410 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:37.071416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:37.071618 (+ 202us) server_negotiation.cc:407] Connection header received
0812 01:12:37.071806 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:37.071810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:37.071885 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:37.071966 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:37.072803 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.073548 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.074822 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.075249 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.075400 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:37.079986 (+ 4586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:44347: BlockingRecv error: recv got EOF from 127.20.151.1:44347 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 12 01:12:37 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:37 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:37.653559 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:37.644155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:54995)
0812 01:12:37.644716 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:37.644732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:37.644842 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:37.645201 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:37.645205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:37.645666 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:37.645840 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:37.645851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.646900 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.646906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:37.648178 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:37.648196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.648923 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.648929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:37.649070 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:37.649720 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:37.649739 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:37.653239 (+ 3500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":305,"threads_started":1}
W20260812 01:12:37.653748 5061 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:37.644299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:54995 (local address 127.20.151.62:39533)
0812 01:12:37.644799 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:37.644804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:37.644822 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:12:37.644943 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:37.644947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:37.645015 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:37.645105 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:37.645966 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.646715 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.648310 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:37.648752 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:37.648847 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:37.653484 (+ 4637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:54995: BlockingRecv error: recv got EOF from 127.20.151.2:54995 (error 108)
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":250,"threads_started":1}
Aug 12 01:12:38 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:38 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:38.041982 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:38.028737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:48597)
0812 01:12:38.029296 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:38.029313 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:38.029481 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:38.029814 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:38.029819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:38.030233 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:38.030403 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:38.030413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.031510 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.031516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:38.032537 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:38.032549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.033269 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.033275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:38.033451 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:38.037756 (+ 4305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:38.037776 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:38.041602 (+ 3826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":254,"threads_started":1}
W20260812 01:12:38.042066 5061 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:38.028965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:48597 (local address 127.20.151.62:39533)
0812 01:12:38.029226 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:38.029232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:38.029388 (+ 156us) server_negotiation.cc:407] Connection header received
0812 01:12:38.029614 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:38.029618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:38.029683 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:38.029767 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:38.030555 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.031359 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.032680 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.033116 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.033227 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:38.041836 (+ 8609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:48597: BlockingRecv error: recv got EOF from 127.20.151.3:48597 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 12 01:12:38 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:38 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:38.090291 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:38.080999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:47731)
0812 01:12:38.081561 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:38.081577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:38.081717 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:38.082077 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:38.082082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:38.082536 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:38.082722 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:38.082733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.083850 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.083856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:38.084864 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:38.084879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.085616 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.085622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:38.085797 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:38.086382 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:38.086400 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:38.090030 (+ 3630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":320,"threads_started":1}
W20260812 01:12:38.090476 5061 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:38.081102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:47731 (local address 127.20.151.62:39533)
0812 01:12:38.081355 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:38.081362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:38.081670 (+ 308us) server_negotiation.cc:407] Connection header received
0812 01:12:38.081842 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:38.081847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:38.081934 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:38.082018 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:38.082874 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.083704 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.085015 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.085465 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.085574 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:38.090228 (+ 4654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:47731: BlockingRecv error: recv got EOF from 127.20.151.1:47731 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 12 01:12:38 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:38 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:38.664122 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:38.654698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:41995)
0812 01:12:38.655801 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:38.655818 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:38.655936 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:38.656315 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:38.656319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:38.656734 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:38.656920 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:38.656930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.657899 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.657903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:38.658828 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:38.658840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.659544 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.659548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:38.659682 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:38.660269 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:38.660287 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:38.663869 (+ 3582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":838,"threads_started":1}
W20260812 01:12:38.664407 5073 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:38.654803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:41995 (local address 127.20.151.62:39533)
0812 01:12:38.655480 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:38.655486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:38.655878 (+ 392us) server_negotiation.cc:407] Connection header received
0812 01:12:38.656049 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:38.656055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:38.656133 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:38.656250 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:38.657062 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.657794 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.658960 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:38.659422 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:38.659519 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:38.664139 (+ 4620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:41995: BlockingRecv error: recv got EOF from 127.20.151.2:41995 (error 108)
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":402,"threads_started":1}
Aug 12 01:12:39 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:39 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:39.052882 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:39.043054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:49157)
0812 01:12:39.043583 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:39.043599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:39.043741 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:39.044049 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:39.044052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:39.044488 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:39.044696 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:39.044706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.045804 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.045808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:39.047197 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:39.047214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.048232 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.048238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:39.048399 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:39.049186 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:39.049208 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:39.052647 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":308,"threads_started":1}
W20260812 01:12:39.053145 5073 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:39.043187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:49157 (local address 127.20.151.62:39533)
0812 01:12:39.043389 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:39.043400 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:39.043695 (+ 295us) server_negotiation.cc:407] Connection header received
0812 01:12:39.043850 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:39.043853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:39.043925 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:39.044035 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:39.044862 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.045679 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.047374 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.048054 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.048203 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:39.052887 (+ 4684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:49157: BlockingRecv error: recv got EOF from 127.20.151.3:49157 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 01:12:39 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:39 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:39.100924 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:39.091259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:41599)
0812 01:12:39.091823 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:39.091839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:39.091975 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:39.092348 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:39.092353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:39.092788 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:39.092969 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:39.092979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.094199 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.094203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:39.095148 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:39.095161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.095964 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.095971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:39.096147 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:39.096808 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:39.096833 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:39.100582 (+ 3749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":281,"threads_started":1}
W20260812 01:12:39.101070 5073 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:39.091355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:41599 (local address 127.20.151.62:39533)
0812 01:12:39.091545 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:39.091551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:39.091923 (+ 372us) server_negotiation.cc:407] Connection header received
0812 01:12:39.092096 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:39.092100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:39.092181 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:39.092267 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:39.093119 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.094046 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.095321 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.095780 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.095894 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:39.100819 (+ 4925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:41599: BlockingRecv error: recv got EOF from 127.20.151.1:41599 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 12 01:12:39 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:39 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:39.674996 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:39.665204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:48089)
0812 01:12:39.665939 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:39.665958 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:39.666080 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:39.666808 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:39.666814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:39.667402 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:39.667598 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:39.667609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.668737 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.668741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:39.669683 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:39.669695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.670427 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.670431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:39.670559 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:39.671144 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:39.671162 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:39.674728 (+ 3566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":400,"threads_started":1}
W20260812 01:12:39.675220 5078 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:39.665338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:48089 (local address 127.20.151.62:39533)
0812 01:12:39.666500 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:39.666504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:39.666520 (+ 16us) server_negotiation.cc:407] Connection header received
0812 01:12:39.666566 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:39.666569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:39.666640 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:39.666747 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:39.667771 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.668607 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.669834 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:39.670278 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:39.670385 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:39.674981 (+ 4596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:48089: BlockingRecv error: recv got EOF from 127.20.151.2:48089 (error 108)
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":911,"threads_started":1}
Aug 12 01:12:40 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:40 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:40.063102 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:40.053885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:55599)
0812 01:12:40.054435 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:40.054454 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:40.054611 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:40.054938 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:40.054941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:40.055405 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:40.055612 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:40.055622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.056668 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.056672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:40.057624 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:40.057635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.058455 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.058462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:40.058658 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:40.059322 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:40.059345 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:40.062836 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":285,"threads_started":1}
W20260812 01:12:40.063313 5078 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:40.054010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:55599 (local address 127.20.151.62:39533)
0812 01:12:40.054255 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:40.054262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:40.054581 (+ 319us) server_negotiation.cc:407] Connection header received
0812 01:12:40.054740 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:40.054744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:40.054810 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:40.054895 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:40.055726 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.056548 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.057775 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.058244 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.058347 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:40.063072 (+ 4725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:55599: BlockingRecv error: recv got EOF from 127.20.151.3:55599 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 12 01:12:40 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:40 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:40.111260 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:40.101986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:51943)
0812 01:12:40.102444 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:40.102462 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:40.102591 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:40.102904 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:40.102908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:40.103370 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:40.103527 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:40.103539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.104675 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.104679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:40.105776 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:40.105791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.106520 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.106524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:40.106653 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:40.107261 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:40.107307 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:40.111011 (+ 3704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
W20260812 01:12:40.111501 5078 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:40.102097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:51943 (local address 127.20.151.62:39533)
0812 01:12:40.102351 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:40.102357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:40.102537 (+ 180us) server_negotiation.cc:407] Connection header received
0812 01:12:40.102705 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:40.102709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:40.102775 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:40.102856 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:40.103684 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.104538 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.105936 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.106392 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.106488 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:40.111214 (+ 4726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:51943: BlockingRecv error: recv got EOF from 127.20.151.1:51943 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 12 01:12:40 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:40 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:40.685372 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:40.676066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:35575)
0812 01:12:40.676552 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:40.676578 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:40.676688 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:40.677181 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:40.677185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:40.677661 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:40.677837 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:40.677848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.678952 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.678956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:40.679938 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:40.679951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.680593 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.680597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:40.680738 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:40.681314 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:40.681332 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:40.685107 (+ 3775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":251,"threads_started":1}
W20260812 01:12:40.685554 5083 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:40.676274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:35575 (local address 127.20.151.62:39533)
0812 01:12:40.676845 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:40.676849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:40.676865 (+ 16us) server_negotiation.cc:407] Connection header received
0812 01:12:40.676913 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:40.676916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:40.677002 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:40.677091 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:40.678014 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.678788 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.680051 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:40.680487 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:40.680578 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:40.685306 (+ 4728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:35575: BlockingRecv error: recv got EOF from 127.20.151.2:35575 (error 108)
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":366,"threads_started":1}
Aug 12 01:12:41 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:41 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.3@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:41.073094 5084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:41.064203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.3:37393)
0812 01:12:41.064695 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:41.064715 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:41.064861 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:41.065228 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:41.065233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:41.065698 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:41.065851 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:41.065861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.067000 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.067005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:41.067949 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:41.067961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.068704 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.068712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:41.068843 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:41.069429 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:41.069446 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:41.072843 (+ 3397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":370,"threads_started":1}
W20260812 01:12:41.073334 5083 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:41.064331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:37393 (local address 127.20.151.62:39533)
0812 01:12:41.064568 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:41.064575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:41.064850 (+ 275us) server_negotiation.cc:407] Connection header received
0812 01:12:41.064996 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:41.065001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:41.065087 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:41.065178 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:41.065975 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.066845 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.068108 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.068565 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.068664 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:41.073082 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.3:37393: BlockingRecv error: recv got EOF from 127.20.151.3:37393 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 01:12:41 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:41 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.1@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:41.121596 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:41.112714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.1:44565)
0812 01:12:41.113266 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:41.113282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:41.113418 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:41.113795 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:41.113800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:41.114205 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:41.114398 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:41.114409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.115471 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.115475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:41.116405 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:41.116416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.117092 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.117098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:41.117282 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:41.117890 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:41.117908 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:41.121222 (+ 3314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":289,"threads_started":1}
W20260812 01:12:41.121695 5083 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:41.112769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:44565 (local address 127.20.151.62:39533)
0812 01:12:41.113121 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:41.113147 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:41.113365 (+ 218us) server_negotiation.cc:407] Connection header received
0812 01:12:41.113540 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:41.113546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:41.113630 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:41.113719 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:41.114547 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.115325 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.116546 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.116954 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.117061 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:41.121451 (+ 4390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.1:44565: BlockingRecv error: recv got EOF from 127.20.151.1:44565 (error 108)
Metrics: {"server-negotiator.queue_time_us":216}
Aug 12 01:12:41 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 01:12:41 dist-test-slave-21b9 krb5kdc[3237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.151.2@KRBTEST.COM for oryx/127.20.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 01:12:41.695871 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:41.686653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39533 (local address 127.20.151.2:57849)
0812 01:12:41.687214 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:41.687231 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:41.687389 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:41.687734 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:41.687737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:41.688137 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:41.688343 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:41.688353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.689462 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.689466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:41.690385 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:41.690396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.691100 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.691104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:41.691237 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:41.691857 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:41.691875 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:41.695598 (+ 3723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.151.62:39533: Server oryx/127.20.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":238,"threads_started":1}
W20260812 01:12:41.696101 5094 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:41.686794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:57849 (local address 127.20.151.62:39533)
0812 01:12:41.687293 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:41.687299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:41.687320 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:12:41.687533 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:41.687537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:41.687613 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:41.687704 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:41.688515 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.689315 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.690535 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.690991 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.691084 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:41.695821 (+ 4737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.151.2:57849: BlockingRecv error: recv got EOF from 127.20.151.2:57849 (error 108)
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":269,"threads_started":1}
I20260812 01:12:41.815551 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 4262
I20260812 01:12:41.829339 5096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:41.816725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.20.151.1:41571)
0812 01:12:41.817365 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:41.817373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:41.817403 (+ 30us) server_negotiation.cc:407] Connection header received
0812 01:12:41.817486 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:41.817492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:41.817741 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:41.817923 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:41.819491 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.820610 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.822013 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.822719 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.822849 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:41.825607 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:41.825618 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:41.825634 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:41.825685 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:41.828726 (+ 3041us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.151.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 01:12:41.829075 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37510: Request ticket server kudu/127.20.151.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":327,"threads_started":1}
W20260812 01:12:41.829471 5096 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37510: Request ticket server kudu/127.20.151.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 01:12:41.829547 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 4262: Not authorized: Client connection negotiation failed: client connection to 127.20.151.1:41571: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.151.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 01:12:41.829748 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 4262
I20260812 01:12:41.862493 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 4520
I20260812 01:12:41.876147 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:41.863671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.20.151.2:39511)
0812 01:12:41.864319 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:41.864327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:41.864361 (+ 34us) server_negotiation.cc:407] Connection header received
0812 01:12:41.864435 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:41.864441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:41.864677 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:41.864838 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:41.866610 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.867837 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.869124 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.869757 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.869867 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:41.872528 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:41.872537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:41.872553 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:41.872596 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:41.875565 (+ 2969us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.151.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 01:12:41.875884 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56760: Request ticket server kudu/127.20.151.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":278,"threads_started":1}
W20260812 01:12:41.876266 5097 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56760: Request ticket server kudu/127.20.151.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 01:12:41.876371 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 4520: Not authorized: Client connection negotiation failed: client connection to 127.20.151.2:39511: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.151.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 01:12:41.876575 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 4520
I20260812 01:12:41.910584 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 4743
I20260812 01:12:41.923693 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:41.911891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.20.151.3:42155)
0812 01:12:41.912499 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:41.912506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:41.912533 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:12:41.912611 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:41.912622 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:41.912882 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:41.913065 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:41.914549 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.915617 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.916890 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:41.917458 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:41.917603 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:41.920278 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:41.920288 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:41.920303 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:41.920345 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:41.922980 (+ 2635us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.151.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 01:12:41.923357 (+ 377us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49122: Request ticket server kudu/127.20.151.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
W20260812 01:12:41.923754 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 4743: Not authorized: Client connection negotiation failed: client connection to 127.20.151.3:42155: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.151.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 01:12:41.923804 5098 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49122: Request ticket server kudu/127.20.151.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 01:12:41.923899 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 4743
I20260812 01:12:41.950377 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 4140
I20260812 01:12:41.951853 4249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:42.082499 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 4140
2026-08-12T01:12:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21896 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T01:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:12:42Z Disabled control of system clock
I20260812 01:12:42.192167 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:40241
--webserver_interface=127.20.151.62
--webserver_port=0
--builtin_ntp_servers=127.20.151.20:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:40241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 01:12:42.557116 5107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:42.557466 5107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:42.557616 5107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:42.565390 5107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:12:42.565541 5107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:42.565616 5107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:42.565711 5107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:12:42.565778 5107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:12:42.575582 5107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:40241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:42.576973 5107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:42.578894 5107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:42.591593 5115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:42.591526 5122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:42.591559 5117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:42.592252 5107 server_base.cc:1034] running on GCE node
I20260812 01:12:42.593091 5107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:42.594527 5107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:42.595736 5107 hybrid_clock.cc:648] HybridClock initialized: now 1786497162595697 us; error 73 us; skew 500 ppm
I20260812 01:12:42.596093 5107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:12:42.598593 5107 webserver.cc:460] Webserver started at http://127.20.151.62:40253/ using document root <none> and password file <none>
I20260812 01:12:42.599360 5107 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:42.599491 5107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:42.599839 5107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:42.602814 5107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "612d7ad17e494fe8962e4aa3a689080c"
format_stamp: "Formatted at 2026-08-12 01:12:42 on dist-test-slave-21b9"
server_key: "7ee31a57815ad787b7143b36e6bcfa8d"
server_key_iv: "44ddd1a8442dd7f58815aa8ca9f05ef2"
server_key_version: "encryptionkey@0"
I20260812 01:12:42.603670 5107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "612d7ad17e494fe8962e4aa3a689080c"
format_stamp: "Formatted at 2026-08-12 01:12:42 on dist-test-slave-21b9"
server_key: "7ee31a57815ad787b7143b36e6bcfa8d"
server_key_iv: "44ddd1a8442dd7f58815aa8ca9f05ef2"
server_key_version: "encryptionkey@0"
I20260812 01:12:42.608924 5107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:12:42.612587 5135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:42.613996 5107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:12:42.614176 5107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "612d7ad17e494fe8962e4aa3a689080c"
format_stamp: "Formatted at 2026-08-12 01:12:42 on dist-test-slave-21b9"
server_key: "7ee31a57815ad787b7143b36e6bcfa8d"
server_key_iv: "44ddd1a8442dd7f58815aa8ca9f05ef2"
server_key_version: "encryptionkey@0"
I20260812 01:12:42.614393 5107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:42.628338 5107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:42.629211 5107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:42.629495 5107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:42.648691 5107 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:40241
I20260812 01:12:42.648729 5219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:40241 every 8 connection(s)
I20260812 01:12:42.650126 5107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:12:42.653592 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 5107
I20260812 01:12:42.653815 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:12:42.654160 5223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:42.654522 21084 external_mini_cluster.cc:1428] Setting key 54c9307dab70fdad9d3e111ccc96d0a7
I20260812 01:12:42.670414 5223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c: Bootstrap starting.
I20260812 01:12:42.670702 5230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:42.656669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.20.151.62:40241)
0812 01:12:42.657944 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:42.657965 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:42.658026 (+ 61us) server_negotiation.cc:407] Connection header received
0812 01:12:42.659770 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:42.659803 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:42.660463 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:42.660980 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:42.661569 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:42.663542 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:42.665020 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:42.666068 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:42.666274 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:42.667521 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:42.667541 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:42.667567 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:42.668358 (+ 791us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:42.668477 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:42.668504 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:42.668638 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:42.668936 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 01:12:42.669303 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":400,"threads_started":1}
I20260812 01:12:42.673923 5223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:42.674934 5223 log.cc:824] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:42.677896 5223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c: No bootstrap required, opened a new log
I20260812 01:12:42.685829 5223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612d7ad17e494fe8962e4aa3a689080c" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40241 } }
I20260812 01:12:42.686259 5223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:42.686337 5223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 612d7ad17e494fe8962e4aa3a689080c, State: Initialized, Role: FOLLOWER
I20260812 01:12:42.686862 5223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612d7ad17e494fe8962e4aa3a689080c" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40241 } }
I20260812 01:12:42.687071 5223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:12:42.687184 5223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:12:42.687341 5223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:42.689551 5223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612d7ad17e494fe8962e4aa3a689080c" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40241 } }
I20260812 01:12:42.690028 5223 leader_election.cc:302] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 612d7ad17e494fe8962e4aa3a689080c; no voters:
I20260812 01:12:42.690627 5223 leader_election.cc:288] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:12:42.690749 5234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:42.691484 5234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [term 1 LEADER]: Becoming Leader. State: Replica: 612d7ad17e494fe8962e4aa3a689080c, State: Running, Role: LEADER
I20260812 01:12:42.692059 5234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612d7ad17e494fe8962e4aa3a689080c" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40241 } }
I20260812 01:12:42.692849 5223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:12:42.695344 5237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 612d7ad17e494fe8962e4aa3a689080c. Latest consensus state: current_term: 1 leader_uuid: "612d7ad17e494fe8962e4aa3a689080c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612d7ad17e494fe8962e4aa3a689080c" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40241 } } }
I20260812 01:12:42.695294 5235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "612d7ad17e494fe8962e4aa3a689080c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612d7ad17e494fe8962e4aa3a689080c" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40241 } } }
I20260812 01:12:42.695755 5237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:42.695829 5235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:42.700716 5244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:12:42.705607 5244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:12:42.715423 5244 catalog_manager.cc:1347] Generated new cluster ID: 0c4db6d98c6b4d4aa093647ca2f6630a
I20260812 01:12:42.715575 5244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:12:42.727324 5244 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:12:42.728693 5244 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:12:42.745807 5244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 612d7ad17e494fe8962e4aa3a689080c: Generated new TSK 0
I20260812 01:12:42.746773 5244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:12:42.759742 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40241
--builtin_ntp_servers=127.20.151.20:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 01:12:43.128199 5262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:43.128515 5262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:43.128628 5262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:43.136210 5262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:43.136339 5262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:43.136489 5262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:12:43.146060 5262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:43.147478 5262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:43.149333 5262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:43.162586 5275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:43.162601 5274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:43.163033 5277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:43.163658 5262 server_base.cc:1034] running on GCE node
I20260812 01:12:43.164299 5262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:43.165306 5262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:43.166577 5262 hybrid_clock.cc:648] HybridClock initialized: now 1786497163166487 us; error 81 us; skew 500 ppm
I20260812 01:12:43.166873 5262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:12:43.175092 5262 webserver.cc:460] Webserver started at http://127.20.151.1:33725/ using document root <none> and password file <none>
I20260812 01:12:43.175853 5262 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:43.175958 5262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:43.176258 5262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:43.179252 5262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "798e0d1dc5dc4bd8b4f6ac1abc013982"
format_stamp: "Formatted at 2026-08-12 01:12:43 on dist-test-slave-21b9"
server_key: "7d673500a6b97df2faea5b4ce6bbc562"
server_key_iv: "e48ecd7baef8a8e45b8ac8dc62d0dbd1"
server_key_version: "encryptionkey@0"
I20260812 01:12:43.180074 5262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "798e0d1dc5dc4bd8b4f6ac1abc013982"
format_stamp: "Formatted at 2026-08-12 01:12:43 on dist-test-slave-21b9"
server_key: "7d673500a6b97df2faea5b4ce6bbc562"
server_key_iv: "e48ecd7baef8a8e45b8ac8dc62d0dbd1"
server_key_version: "encryptionkey@0"
I20260812 01:12:43.185174 5262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:12:43.188670 5292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:43.189939 5262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:12:43.190101 5262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "798e0d1dc5dc4bd8b4f6ac1abc013982"
format_stamp: "Formatted at 2026-08-12 01:12:43 on dist-test-slave-21b9"
server_key: "7d673500a6b97df2faea5b4ce6bbc562"
server_key_iv: "e48ecd7baef8a8e45b8ac8dc62d0dbd1"
server_key_version: "encryptionkey@0"
I20260812 01:12:43.190294 5262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:43.216306 5262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:43.217249 5262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:43.217614 5262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:43.218640 5262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:43.220355 5262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:43.220448 5262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:43.220534 5262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:43.220595 5262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:43.253166 5262 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:35779
I20260812 01:12:43.253247 5497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:35779 every 8 connection(s)
I20260812 01:12:43.254498 5262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:12:43.264835 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 5262
I20260812 01:12:43.265070 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:12:43.265897 21084 external_mini_cluster.cc:1428] Setting key 574d1f2a8c9357d8d0c07166cc91ef48
I20260812 01:12:43.267468 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:43.257112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:47653 (local address 127.20.151.62:40241)
0812 01:12:43.257576 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:43.257581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:43.258812 (+ 1231us) server_negotiation.cc:407] Connection header received
0812 01:12:43.260682 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:43.260686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:43.260768 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:43.260870 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:43.261954 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:43.262715 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:43.264363 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:43.264815 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:43.264916 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:43.266625 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:43.266631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:43.266634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:43.266653 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:43.266702 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:43.266719 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:43.266811 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:43.267038 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 01:12:43.267118 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":201,"threads_started":1}
I20260812 01:12:43.268806 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:43.257700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40241 (local address 127.20.151.1:47653)
0812 01:12:43.258659 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:43.258695 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:43.260377 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:43.261048 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:43.261055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:43.261148 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:43.261758 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:43.261775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:43.262895 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:43.262899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:43.264205 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:43.264218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:43.265076 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:43.265084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:43.265409 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:43.266194 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:12:43.266198 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:43.266252 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:12:43.266362 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:43.266834 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:43.266844 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:43.266850 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 01:12:43.267032 (+ 182us) client_negotiation.cc:240] Negotiation successful
0812 01:12:43.267459 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":272,"threads_started":1}
I20260812 01:12:43.269158 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40241
--builtin_ntp_servers=127.20.151.20:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 01:12:43.270388 5499 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40241
I20260812 01:12:43.270709 5499 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:43.271504 5499 heartbeater.cc:511] Master 127.20.151.62:40241 requested a full tablet report, sending...
I20260812 01:12:43.273676 5159 ts_manager.cc:194] Registered new tserver with Master: 798e0d1dc5dc4bd8b4f6ac1abc013982 (127.20.151.1:35779)
I20260812 01:12:43.275609 5159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.1:47653
W20260812 01:12:43.625473 5511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:43.625782 5511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:43.625912 5511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:43.633265 5511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:43.633378 5511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:43.633531 5511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:12:43.642930 5511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:43.644313 5511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:43.646227 5511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:43.661124 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:43.661055 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:43.661646 5511 server_base.cc:1034] running on GCE node
W20260812 01:12:43.661737 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:43.662596 5511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:43.663851 5511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:43.665072 5511 hybrid_clock.cc:648] HybridClock initialized: now 1786497163665013 us; error 53 us; skew 500 ppm
I20260812 01:12:43.665398 5511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:12:43.667897 5511 webserver.cc:460] Webserver started at http://127.20.151.2:43653/ using document root <none> and password file <none>
I20260812 01:12:43.668617 5511 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:43.668725 5511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:43.669049 5511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:43.672056 5511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "b38436f2ab8248c4b85d3fca0b0d3a02"
format_stamp: "Formatted at 2026-08-12 01:12:43 on dist-test-slave-21b9"
server_key: "4056219070517d3d5c19426408c10fc7"
server_key_iv: "2e25f03151478d67860d292cacddc61b"
server_key_version: "encryptionkey@0"
I20260812 01:12:43.672900 5511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "b38436f2ab8248c4b85d3fca0b0d3a02"
format_stamp: "Formatted at 2026-08-12 01:12:43 on dist-test-slave-21b9"
server_key: "4056219070517d3d5c19426408c10fc7"
server_key_iv: "2e25f03151478d67860d292cacddc61b"
server_key_version: "encryptionkey@0"
I20260812 01:12:43.678308 5511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 01:12:43.682073 5529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:43.683580 5511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:12:43.683802 5511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "b38436f2ab8248c4b85d3fca0b0d3a02"
format_stamp: "Formatted at 2026-08-12 01:12:43 on dist-test-slave-21b9"
server_key: "4056219070517d3d5c19426408c10fc7"
server_key_iv: "2e25f03151478d67860d292cacddc61b"
server_key_version: "encryptionkey@0"
I20260812 01:12:43.684022 5511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:43.706132 5511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:43.707031 5511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:43.707352 5511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:43.708477 5511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:43.710083 5511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:43.710184 5511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:43.710294 5511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:43.710366 5511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:43.741873 5511 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:33555
I20260812 01:12:43.741952 5720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:33555 every 8 connection(s)
I20260812 01:12:43.743148 5511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:12:43.744894 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 5511
I20260812 01:12:43.745064 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:12:43.745501 21084 external_mini_cluster.cc:1428] Setting key 6a7c0bba5a7b57177633684e22eb25ed
I20260812 01:12:43.748253 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40241
--builtin_ntp_servers=127.20.151.20:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 01:12:43.756127 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:43.745831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:51839 (local address 127.20.151.62:40241)
0812 01:12:43.746338 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:43.746344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:43.747709 (+ 1365us) server_negotiation.cc:407] Connection header received
0812 01:12:43.749542 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:43.749548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:43.749638 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:43.749743 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:43.750843 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:43.751812 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:43.753381 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:43.753907 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:43.754026 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:43.755396 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:43.755402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:43.755406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:43.755427 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:43.755465 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:43.755479 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:43.755561 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:43.755737 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 01:12:43.755829 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366}
I20260812 01:12:43.757082 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:43.746440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40241 (local address 127.20.151.2:51839)
0812 01:12:43.747539 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:43.747594 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:43.749273 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:43.749971 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:43.749979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:43.750066 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:43.750674 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:43.750693 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:43.751988 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:43.751992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:43.753236 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:43.753250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:43.754147 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:43.754152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:43.754372 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:43.755059 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:12:43.755062 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:43.755103 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:12:43.755189 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:43.755559 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:43.755567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:43.755572 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 01:12:43.755708 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 01:12:43.756002 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":316,"threads_started":1}
I20260812 01:12:43.758399 5721 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40241
I20260812 01:12:43.758735 5721 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:43.759642 5721 heartbeater.cc:511] Master 127.20.151.62:40241 requested a full tablet report, sending...
I20260812 01:12:43.761173 5159 ts_manager.cc:194] Registered new tserver with Master: b38436f2ab8248c4b85d3fca0b0d3a02 (127.20.151.2:33555)
I20260812 01:12:43.762035 5159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.2:51839
W20260812 01:12:44.098179 5731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:44.098487 5731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:44.098624 5731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:44.106109 5731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:44.106232 5731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:44.106371 5731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:12:44.115797 5731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:44.117192 5731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:44.119095 5731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:44.132663 5740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:44.132601 5747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:44.132633 5741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:44.133452 5731 server_base.cc:1034] running on GCE node
I20260812 01:12:44.134079 5731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:44.135070 5731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:44.136296 5731 hybrid_clock.cc:648] HybridClock initialized: now 1786497164136212 us; error 70 us; skew 500 ppm
I20260812 01:12:44.136643 5731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:12:44.139048 5731 webserver.cc:460] Webserver started at http://127.20.151.3:45971/ using document root <none> and password file <none>
I20260812 01:12:44.139810 5731 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:44.139930 5731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:44.140244 5731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:44.143160 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "b17bf7717e094cf8bd298f374909aab5"
format_stamp: "Formatted at 2026-08-12 01:12:44 on dist-test-slave-21b9"
server_key: "86e6cd026a03c074e8619d373a043e5f"
server_key_iv: "da880c9b423a6233224a571c0e8a379a"
server_key_version: "encryptionkey@0"
I20260812 01:12:44.143994 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "b17bf7717e094cf8bd298f374909aab5"
format_stamp: "Formatted at 2026-08-12 01:12:44 on dist-test-slave-21b9"
server_key: "86e6cd026a03c074e8619d373a043e5f"
server_key_iv: "da880c9b423a6233224a571c0e8a379a"
server_key_version: "encryptionkey@0"
I20260812 01:12:44.149227 5731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:12:44.152684 5757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:44.153966 5731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:12:44.154124 5731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "b17bf7717e094cf8bd298f374909aab5"
format_stamp: "Formatted at 2026-08-12 01:12:44 on dist-test-slave-21b9"
server_key: "86e6cd026a03c074e8619d373a043e5f"
server_key_iv: "da880c9b423a6233224a571c0e8a379a"
server_key_version: "encryptionkey@0"
I20260812 01:12:44.154333 5731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:44.180099 5731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:44.180989 5731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:44.181337 5731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:44.182404 5731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:44.183974 5731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:44.184065 5731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:44.184141 5731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:44.184196 5731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:44.215133 5731 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:42995
I20260812 01:12:44.215231 5944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:42995 every 8 connection(s)
I20260812 01:12:44.216447 5731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:12:44.220546 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 5731
I20260812 01:12:44.220752 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:12:44.221194 21084 external_mini_cluster.cc:1428] Setting key accce7284029ea5ec24bb71d102e1475
I20260812 01:12:44.229055 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:44.219041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:48061 (local address 127.20.151.62:40241)
0812 01:12:44.219324 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:44.219330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:44.220747 (+ 1417us) server_negotiation.cc:407] Connection header received
0812 01:12:44.222810 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:44.222816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:44.222905 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:44.223021 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:44.224093 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:44.224943 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:44.226471 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:44.226886 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:44.226980 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:44.228265 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:44.228270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:44.228273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:44.228292 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:44.228324 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:44.228335 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:44.228405 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:44.228703 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 01:12:44.228785 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 01:12:44.229988 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:44.219643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40241 (local address 127.20.151.3:48061)
0812 01:12:44.220560 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:44.220594 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:44.222533 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:44.223237 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:44.223245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:44.223372 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:44.223959 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:44.223977 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:44.225091 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:44.225094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:44.226324 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:44.226336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:44.227069 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:44.227074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:44.227305 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:44.227945 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:12:44.227950 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:44.227997 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:12:44.228079 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:44.228456 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:44.228485 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:44.228493 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 01:12:44.228647 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 01:12:44.228941 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":282,"threads_started":1}
I20260812 01:12:44.231317 5945 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40241
I20260812 01:12:44.231617 5945 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:44.232339 5945 heartbeater.cc:511] Master 127.20.151.62:40241 requested a full tablet report, sending...
I20260812 01:12:44.233882 5159 ts_manager.cc:194] Registered new tserver with Master: b17bf7717e094cf8bd298f374909aab5 (127.20.151.3:42995)
I20260812 01:12:44.234776 5159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.3:48061
I20260812 01:12:44.236828 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 01:12:44.242995 5506 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:44.241363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.20.151.62:40241)
0812 01:12:44.241555 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:44.241561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:44.242069 (+ 508us) server_negotiation.cc:407] Connection header received
0812 01:12:44.242186 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:44.242190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:44.242256 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:44.242343 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:44.242780 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39090: BlockingRecv error: recv got EOF from 127.0.0.1:39090 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:12:44.243634 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 5262
I20260812 01:12:44.251721 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:44.244663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.20.151.1:35779)
0812 01:12:44.245231 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:44.245239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:44.245265 (+ 26us) server_negotiation.cc:407] Connection header received
0812 01:12:44.245319 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:44.245325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:44.245527 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:44.245693 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:44.246183 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:44.247354 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:44.248691 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:44.249254 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:44.249358 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:44.250345 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:44.250354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:44.250366 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:44.250952 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:44.251008 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:44.251028 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:44.251146 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:44.251253 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 01:12:44.251407 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":302,"threads_started":1}
I20260812 01:12:44.252976 5491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:44.334992 5499 heartbeater.cc:503] Master 127.20.151.62:40241 was elected leader, sending a full tablet report...
I20260812 01:12:44.336303 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 5262
W20260812 01:12:44.361528 5137 connection.cc:535] server connection from 127.20.151.1:47653 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 01:12:44.361831 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 5511
I20260812 01:12:44.370324 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:44.362982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.20.151.2:33555)
0812 01:12:44.363619 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:44.363627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:44.363656 (+ 29us) server_negotiation.cc:407] Connection header received
0812 01:12:44.363719 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:44.363726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:44.363944 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:44.364077 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:44.364504 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:44.365566 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:44.366892 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:44.367606 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:44.367710 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:44.368767 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:44.368776 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:44.368790 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:44.369421 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:44.369471 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:44.369493 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:44.369596 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:44.369830 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 01:12:44.369979 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":306,"threads_started":1}
I20260812 01:12:44.371646 5712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:44.454360 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 5511
I20260812 01:12:44.487694 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 5731
I20260812 01:12:44.496454 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:44.488779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.20.151.3:42995)
0812 01:12:44.489445 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:44.489453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:44.489483 (+ 30us) server_negotiation.cc:407] Connection header received
0812 01:12:44.489545 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:44.489551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:44.489801 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:44.489947 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:44.490429 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:44.491536 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:44.492778 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:44.493428 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:44.493553 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:44.494730 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:44.494740 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:44.494755 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:44.495524 (+ 769us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:44.495609 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:44.495633 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:44.495751 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:44.495945 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 01:12:44.496106 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":360,"threads_started":1}
I20260812 01:12:44.497774 5939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:44.580658 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 5731
I20260812 01:12:44.615092 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 5107
I20260812 01:12:44.616428 5213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:44.731369 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 5107
2026-08-12T01:12:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2636 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T01:12:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:12:44Z Disabled control of system clock
I20260812 01:12:44.830681 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46613
--webserver_interface=127.20.151.62
--webserver_port=0
--builtin_ntp_servers=127.20.151.20:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:46613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 01:12:45.184966 5975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:45.185292 5975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:45.185395 5975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:45.192955 5975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:12:45.193092 5975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:45.193131 5975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:45.193189 5975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:12:45.193235 5975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:12:45.203015 5975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:46613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:45.204460 5975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:45.206336 5975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:45.218995 5982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:45.219079 5984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:45.219022 5981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:45.219828 5975 server_base.cc:1034] running on GCE node
I20260812 01:12:45.220726 5975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:45.222242 5975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:45.223452 5975 hybrid_clock.cc:648] HybridClock initialized: now 1786497165223407 us; error 32 us; skew 500 ppm
I20260812 01:12:45.223892 5975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:12:45.226223 5975 webserver.cc:460] Webserver started at http://127.20.151.62:46551/ using document root <none> and password file <none>
I20260812 01:12:45.226933 5975 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:45.227029 5975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:45.227423 5975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:45.232223 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "bde99922a3024572a305c6cdd12d2ef6"
format_stamp: "Formatted at 2026-08-12 01:12:45 on dist-test-slave-21b9"
server_key: "3261ef8c16d7cb47bba38c76999228de"
server_key_iv: "3586ed61174f9695f62dc324857222b8"
server_key_version: "encryptionkey@0"
I20260812 01:12:45.233006 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "bde99922a3024572a305c6cdd12d2ef6"
format_stamp: "Formatted at 2026-08-12 01:12:45 on dist-test-slave-21b9"
server_key: "3261ef8c16d7cb47bba38c76999228de"
server_key_iv: "3586ed61174f9695f62dc324857222b8"
server_key_version: "encryptionkey@0"
I20260812 01:12:45.238361 5975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:12:45.241968 5997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:45.243337 5975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 01:12:45.243510 5975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "bde99922a3024572a305c6cdd12d2ef6"
format_stamp: "Formatted at 2026-08-12 01:12:45 on dist-test-slave-21b9"
server_key: "3261ef8c16d7cb47bba38c76999228de"
server_key_iv: "3586ed61174f9695f62dc324857222b8"
server_key_version: "encryptionkey@0"
I20260812 01:12:45.243722 5975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:45.249511 5975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:45.250276 5975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:45.250567 5975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:45.269460 5975 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:46613
I20260812 01:12:45.269495 6074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:46613 every 8 connection(s)
I20260812 01:12:45.270898 5975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:12:45.272400 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 5975
I20260812 01:12:45.272614 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:12:45.273505 21084 external_mini_cluster.cc:1428] Setting key 184bc5a63cfde16d9189a65cb3b802f4
I20260812 01:12:45.275945 6076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:45.283311 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:45.275735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.20.151.62:46613)
0812 01:12:45.276811 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:45.276826 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:45.276879 (+ 53us) server_negotiation.cc:407] Connection header received
0812 01:12:45.279041 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:45.279071 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:45.279656 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:45.280135 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:45.281196 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:45.281203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:45.281220 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:45.281238 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:45.281331 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:45.281353 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:45.281583 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:45.281725 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 01:12:45.282020 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":340,"threads_started":1}
I20260812 01:12:45.289485 6076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6: Bootstrap starting.
I20260812 01:12:45.292901 6076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:45.294031 6076 log.cc:824] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:45.297135 6076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6: No bootstrap required, opened a new log
I20260812 01:12:45.304670 6076 raft_consensus.cc:348] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde99922a3024572a305c6cdd12d2ef6" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46613 } }
I20260812 01:12:45.305094 6076 raft_consensus.cc:374] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:45.305209 6076 raft_consensus.cc:729] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde99922a3024572a305c6cdd12d2ef6, State: Initialized, Role: FOLLOWER
I20260812 01:12:45.305810 6076 consensus_queue.cc:260] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde99922a3024572a305c6cdd12d2ef6" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46613 } }
I20260812 01:12:45.306046 6076 raft_consensus.cc:388] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:12:45.306185 6076 raft_consensus.cc:482] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:12:45.306337 6076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:45.308715 6076 raft_consensus.cc:504] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde99922a3024572a305c6cdd12d2ef6" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46613 } }
I20260812 01:12:45.309270 6076 leader_election.cc:302] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bde99922a3024572a305c6cdd12d2ef6; no voters:
I20260812 01:12:45.309917 6076 leader_election.cc:288] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:12:45.310060 6087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:45.310781 6087 raft_consensus.cc:686] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [term 1 LEADER]: Becoming Leader. State: Replica: bde99922a3024572a305c6cdd12d2ef6, State: Running, Role: LEADER
I20260812 01:12:45.311455 6087 consensus_queue.cc:237] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde99922a3024572a305c6cdd12d2ef6" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46613 } }
I20260812 01:12:45.312237 6076 sys_catalog.cc:564] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:12:45.314662 6089 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde99922a3024572a305c6cdd12d2ef6. Latest consensus state: current_term: 1 leader_uuid: "bde99922a3024572a305c6cdd12d2ef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde99922a3024572a305c6cdd12d2ef6" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46613 } } }
I20260812 01:12:45.315395 6089 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:45.315189 6088 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bde99922a3024572a305c6cdd12d2ef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde99922a3024572a305c6cdd12d2ef6" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46613 } } }
I20260812 01:12:45.315686 6088 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6 [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:45.318145 6101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:12:45.323124 6101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:12:45.332491 6101 catalog_manager.cc:1347] Generated new cluster ID: 9adbf1a596ae43c3995339216e898722
I20260812 01:12:45.332623 6101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:12:45.360066 6101 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:12:45.360491 6101 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:12:45.370600 6101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bde99922a3024572a305c6cdd12d2ef6: Generated new TSK 0
I20260812 01:12:45.371516 6101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:12:45.381944 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46613
--builtin_ntp_servers=127.20.151.20:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 01:12:45.741043 6114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:45.741405 6114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:45.741555 6114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:45.749234 6114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:45.749346 6114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:45.749490 6114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:12:45.758885 6114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:45.760237 6114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:45.762140 6114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:45.775748 6123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:45.775811 6126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:45.775844 6122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:45.776732 6114 server_base.cc:1034] running on GCE node
I20260812 01:12:45.777350 6114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:45.778412 6114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:45.779718 6114 hybrid_clock.cc:648] HybridClock initialized: now 1786497165779650 us; error 54 us; skew 500 ppm
I20260812 01:12:45.780054 6114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:12:45.782508 6114 webserver.cc:460] Webserver started at http://127.20.151.1:45637/ using document root <none> and password file <none>
I20260812 01:12:45.783231 6114 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:45.783372 6114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:45.783720 6114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:45.786682 6114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "6dd5a867ba1845a4b1b8cc3ef700c4c9"
format_stamp: "Formatted at 2026-08-12 01:12:45 on dist-test-slave-21b9"
server_key: "de3694a7ecdddbe07faff0a866ee49cb"
server_key_iv: "b54564296e4833c11656064a36b46250"
server_key_version: "encryptionkey@0"
I20260812 01:12:45.787523 6114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "6dd5a867ba1845a4b1b8cc3ef700c4c9"
format_stamp: "Formatted at 2026-08-12 01:12:45 on dist-test-slave-21b9"
server_key: "de3694a7ecdddbe07faff0a866ee49cb"
server_key_iv: "b54564296e4833c11656064a36b46250"
server_key_version: "encryptionkey@0"
I20260812 01:12:45.792863 6114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:12:45.796459 6136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:45.797864 6114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:12:45.798024 6114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "6dd5a867ba1845a4b1b8cc3ef700c4c9"
format_stamp: "Formatted at 2026-08-12 01:12:45 on dist-test-slave-21b9"
server_key: "de3694a7ecdddbe07faff0a866ee49cb"
server_key_iv: "b54564296e4833c11656064a36b46250"
server_key_version: "encryptionkey@0"
I20260812 01:12:45.798243 6114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:45.804821 6114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:45.805807 6114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:45.806087 6114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:45.807083 6114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:45.808627 6114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:45.808720 6114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:45.808832 6114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:45.808924 6114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:45.839138 6114 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:33119
I20260812 01:12:45.839203 6327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:33119 every 8 connection(s)
I20260812 01:12:45.840492 6114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:12:45.845800 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 6114
I20260812 01:12:45.845968 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:12:45.846616 21084 external_mini_cluster.cc:1428] Setting key f41cbe8dc6f7f1ca5585da824cc463e1
I20260812 01:12:45.849342 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:45.843147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:39865 (local address 127.20.151.62:46613)
0812 01:12:45.843623 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:45.843628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:45.844942 (+ 1314us) server_negotiation.cc:407] Connection header received
0812 01:12:45.846906 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:45.846913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:45.846991 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:45.847131 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:45.848564 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:45.848570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:45.848574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:45.848592 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:45.848636 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:45.848648 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:45.848770 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:45.848943 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 01:12:45.849006 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":240,"threads_started":1}
I20260812 01:12:45.849762 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46613
--builtin_ntp_servers=127.20.151.20:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 01:12:45.850240 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:45.843750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46613 (local address 127.20.151.1:39865)
0812 01:12:45.844780 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:45.844824 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:45.846614 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:45.847390 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:45.847402 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:45.847518 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:45.848230 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:12:45.848234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:45.848283 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:12:45.848400 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:45.848751 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:45.848761 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:45.848767 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 01:12:45.848991 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 01:12:45.849218 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":303,"threads_started":1}
I20260812 01:12:45.851799 6329 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46613
I20260812 01:12:45.852110 6329 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:45.852706 6329 heartbeater.cc:511] Master 127.20.151.62:46613 requested a full tablet report, sending...
I20260812 01:12:45.854795 6025 ts_manager.cc:194] Registered new tserver with Master: 6dd5a867ba1845a4b1b8cc3ef700c4c9 (127.20.151.1:33119)
W20260812 01:12:46.204154 6335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:46.204483 6335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:46.204628 6335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:46.212118 6335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:46.212255 6335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:46.212432 6335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:12:46.221976 6335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:46.223560 6335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:46.225513 6335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:46.238788 6350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:46.238744 6352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:46.238827 6355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:46.239925 6335 server_base.cc:1034] running on GCE node
I20260812 01:12:46.240581 6335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:46.241634 6335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:46.242905 6335 hybrid_clock.cc:648] HybridClock initialized: now 1786497166242854 us; error 39 us; skew 500 ppm
I20260812 01:12:46.243223 6335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:12:46.245633 6335 webserver.cc:460] Webserver started at http://127.20.151.2:34011/ using document root <none> and password file <none>
I20260812 01:12:46.246340 6335 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:46.246438 6335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:46.246778 6335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:46.249840 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "65bfae5dc792466cbd117c8e766fb43c"
format_stamp: "Formatted at 2026-08-12 01:12:46 on dist-test-slave-21b9"
server_key: "3230a64526e274dbb587604b83161402"
server_key_iv: "f28adfb8de5dd7bb5b7eb362c4f75455"
server_key_version: "encryptionkey@0"
I20260812 01:12:46.250643 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "65bfae5dc792466cbd117c8e766fb43c"
format_stamp: "Formatted at 2026-08-12 01:12:46 on dist-test-slave-21b9"
server_key: "3230a64526e274dbb587604b83161402"
server_key_iv: "f28adfb8de5dd7bb5b7eb362c4f75455"
server_key_version: "encryptionkey@0"
I20260812 01:12:46.256044 6335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 01:12:46.259567 6363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:46.260975 6335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:12:46.261135 6335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "65bfae5dc792466cbd117c8e766fb43c"
format_stamp: "Formatted at 2026-08-12 01:12:46 on dist-test-slave-21b9"
server_key: "3230a64526e274dbb587604b83161402"
server_key_iv: "f28adfb8de5dd7bb5b7eb362c4f75455"
server_key_version: "encryptionkey@0"
I20260812 01:12:46.261338 6335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:46.266939 6335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:46.267794 6335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:46.268042 6335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:46.269040 6335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:46.270514 6335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:46.270618 6335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:46.270722 6335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:46.270789 6335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:46.303356 6335 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:45449
I20260812 01:12:46.303406 6548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:45449 every 8 connection(s)
I20260812 01:12:46.304708 6335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:12:46.312228 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 6335
I20260812 01:12:46.312467 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:12:46.312834 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:46.307326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:55965 (local address 127.20.151.62:46613)
0812 01:12:46.307538 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:46.307545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:46.308921 (+ 1376us) server_negotiation.cc:407] Connection header received
0812 01:12:46.310629 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:46.310634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:46.310702 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:46.310790 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:46.312055 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:46.312061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:46.312065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:46.312086 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:46.312128 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:46.312143 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:46.312272 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:46.312444 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 01:12:46.312535 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:12:46.312974 21084 external_mini_cluster.cc:1428] Setting key 181a8c6f0cc85ef19fad4a61a93c3e28
I20260812 01:12:46.313844 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:46.307855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46613 (local address 127.20.151.2:55965)
0812 01:12:46.308765 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:46.308801 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:46.310359 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:46.310982 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:46.310989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:46.311096 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:46.311760 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:12:46.311762 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:46.311800 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:12:46.311886 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:46.312239 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:46.312245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:46.312250 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 01:12:46.312515 (+ 265us) client_negotiation.cc:240] Negotiation successful
0812 01:12:46.312768 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":266,"threads_started":1}
I20260812 01:12:46.315639 6549 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46613
I20260812 01:12:46.315981 6549 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:46.316066 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46613
--builtin_ntp_servers=127.20.151.20:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 01:12:46.316746 6549 heartbeater.cc:511] Master 127.20.151.62:46613 requested a full tablet report, sending...
I20260812 01:12:46.318326 6023 ts_manager.cc:194] Registered new tserver with Master: 65bfae5dc792466cbd117c8e766fb43c (127.20.151.2:45449)
W20260812 01:12:46.662931 6554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:46.663235 6554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:46.663383 6554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:46.670866 6554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:46.670977 6554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:46.671089 6554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:12:46.680263 6554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:46.681533 6554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:46.683296 6554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:46.696686 6564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:46.696735 6561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:46.696617 6562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:46.697800 6554 server_base.cc:1034] running on GCE node
I20260812 01:12:46.698472 6554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:46.699481 6554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:46.700719 6554 hybrid_clock.cc:648] HybridClock initialized: now 1786497166700662 us; error 44 us; skew 500 ppm
I20260812 01:12:46.701078 6554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:12:46.703516 6554 webserver.cc:460] Webserver started at http://127.20.151.3:36905/ using document root <none> and password file <none>
I20260812 01:12:46.704241 6554 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:46.704352 6554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:46.704675 6554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:46.707799 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "f2dadd66c7cb45f2aaeb550b8d2cf90f"
format_stamp: "Formatted at 2026-08-12 01:12:46 on dist-test-slave-21b9"
server_key: "3c0e36a313d49a72660c576b36b88d0b"
server_key_iv: "4a114d85d045ca89e20916c042f798e5"
server_key_version: "encryptionkey@0"
I20260812 01:12:46.708699 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "f2dadd66c7cb45f2aaeb550b8d2cf90f"
format_stamp: "Formatted at 2026-08-12 01:12:46 on dist-test-slave-21b9"
server_key: "3c0e36a313d49a72660c576b36b88d0b"
server_key_iv: "4a114d85d045ca89e20916c042f798e5"
server_key_version: "encryptionkey@0"
I20260812 01:12:46.714201 6554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 01:12:46.717904 6572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:46.719306 6554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 01:12:46.719473 6554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "f2dadd66c7cb45f2aaeb550b8d2cf90f"
format_stamp: "Formatted at 2026-08-12 01:12:46 on dist-test-slave-21b9"
server_key: "3c0e36a313d49a72660c576b36b88d0b"
server_key_iv: "4a114d85d045ca89e20916c042f798e5"
server_key_version: "encryptionkey@0"
I20260812 01:12:46.719696 6554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:46.725445 6554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:46.726240 6554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:46.726500 6554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:46.727562 6554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:46.728955 6554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:46.729053 6554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:46.729183 6554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:46.729255 6554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:46.760438 6554 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:33949
I20260812 01:12:46.760493 6753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:33949 every 8 connection(s)
I20260812 01:12:46.761756 6554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:12:46.768095 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 6554
I20260812 01:12:46.768285 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:12:46.768878 21084 external_mini_cluster.cc:1428] Setting key 16241c8939feb0584c267d411c92a721
I20260812 01:12:46.770457 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:46.764527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:37005 (local address 127.20.151.62:46613)
0812 01:12:46.764745 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:46.764751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:46.766193 (+ 1442us) server_negotiation.cc:407] Connection header received
0812 01:12:46.767979 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:46.767983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:46.768053 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:46.768140 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:46.769715 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:46.769724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:46.769728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:46.769746 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:46.769783 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:46.769795 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:46.769894 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:46.770121 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 01:12:46.770180 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:12:46.771595 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:46.765071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46613 (local address 127.20.151.3:37005)
0812 01:12:46.766033 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:46.766069 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:46.767670 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:46.768424 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:46.768436 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:46.768560 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:46.769367 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:12:46.769371 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:46.769425 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:12:46.769527 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:46.769895 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:46.769907 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:46.769915 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 01:12:46.770169 (+ 254us) client_negotiation.cc:240] Negotiation successful
0812 01:12:46.770449 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":255,"threads_started":1}
I20260812 01:12:46.773294 6754 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46613
I20260812 01:12:46.773576 6754 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:46.774168 6754 heartbeater.cc:511] Master 127.20.151.62:46613 requested a full tablet report, sending...
I20260812 01:12:46.775585 6023 ts_manager.cc:194] Registered new tserver with Master: f2dadd66c7cb45f2aaeb550b8d2cf90f (127.20.151.3:33949)
I20260812 01:12:46.784931 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 01:12:46.791138 6332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:12:46.789495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.20.151.62:46613)
0812 01:12:46.789697 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:46.789704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:46.790199 (+ 495us) server_negotiation.cc:407] Connection header received
0812 01:12:46.790333 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:46.790336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:46.790390 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:46.790470 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:46.790897 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51502: BlockingRecv error: recv got EOF from 127.0.0.1:51502 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 01:12:46.791693 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 6114
I20260812 01:12:46.795233 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:46.792719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.20.151.1:33119)
0812 01:12:46.793273 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:46.793280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:46.793307 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:12:46.793366 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:46.793373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:46.793627 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:46.793757 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:46.794632 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:46.794639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:46.794651 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:46.794669 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:46.794716 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:46.794729 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:46.794835 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:46.794890 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 01:12:46.794981 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":274,"threads_started":1}
I20260812 01:12:46.796514 6322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:46.878177 6329 heartbeater.cc:503] Master 127.20.151.62:46613 was elected leader, sending a full tablet report...
I20260812 01:12:46.879475 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 6114
W20260812 01:12:46.904330 6003 connection.cc:535] server connection from 127.20.151.1:39865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 01:12:46.904635 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 6335
I20260812 01:12:46.908233 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:46.905644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.20.151.2:45449)
0812 01:12:46.906154 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:46.906169 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:46.906196 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:12:46.906260 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:46.906266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:46.906482 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:46.906614 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:46.907629 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:46.907636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:46.907647 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:46.907667 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:46.907710 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:46.907724 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:46.907836 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:46.907891 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 01:12:46.907969 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":223,"threads_started":1}
I20260812 01:12:46.909828 6543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:46.992442 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 6335
I20260812 01:12:47.013921 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 6554
I20260812 01:12:47.018477 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:47.014954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.20.151.3:33949)
0812 01:12:47.015697 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:47.015704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:47.015740 (+ 36us) server_negotiation.cc:407] Connection header received
0812 01:12:47.015819 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:47.015825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:47.016063 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:47.016222 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:47.017551 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:47.017562 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:47.017578 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:12:47.017605 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:47.017652 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:12:47.017670 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:47.017856 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:47.017955 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 01:12:47.018065 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":386,"threads_started":1}
I20260812 01:12:47.019822 6744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:47.102447 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 6554
I20260812 01:12:47.132282 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 5975
I20260812 01:12:47.133601 6068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:47.240332 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 5975
2026-08-12T01:12:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2510 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:12:47 dist-test-slave-21b9 krb5kdc[6782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:12:47 dist-test-slave-21b9 krb5kdc[6782](info): set up 2 sockets
Aug 12 01:12:47 dist-test-slave-21b9 krb5kdc[6782](info): commencing operation
krb5kdc: starting...
W20260812 01:12:48.537457 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.004s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:12:48 dist-test-slave-21b9 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:12:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:12:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:12:48.867151 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:45737
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:45737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:49.219310 6800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:49.219647 6800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:49.219822 6800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:49.227473 6800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:12:49.227610 6800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:49.227690 6800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:49.227772 6800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:12:49.227845 6800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:12:49.238380 6800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:45737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:45737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:49.239795 6800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:49.241102 6800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:49.253008 6811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:49.252955 6812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:49.252928 6818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:49.253845 6800 server_base.cc:1034] running on GCE node
I20260812 01:12:49.254729 6800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:49.256294 6800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:49.257501 6800 hybrid_clock.cc:648] HybridClock initialized: now 1786497169257445 us; error 55 us; skew 500 ppm
I20260812 01:12:49.257853 6800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:49 dist-test-slave-21b9 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:49.264349 6800 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:12:49.265976 6800 webserver.cc:460] Webserver started at http://127.20.151.62:40297/ using document root <none> and password file <none>
I20260812 01:12:49.266731 6800 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:49.266865 6800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:49.267171 6800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:49.270397 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "38b10e813657450087b33d374651ffcc"
format_stamp: "Formatted at 2026-08-12 01:12:49 on dist-test-slave-21b9"
server_key: "21d27da2093be3c9e67fa507cb554e2f"
server_key_iv: "412bade3011c9d5ed6bd4c1f1b7aea7b"
server_key_version: "encryptionkey@0"
I20260812 01:12:49.271211 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "38b10e813657450087b33d374651ffcc"
format_stamp: "Formatted at 2026-08-12 01:12:49 on dist-test-slave-21b9"
server_key: "21d27da2093be3c9e67fa507cb554e2f"
server_key_iv: "412bade3011c9d5ed6bd4c1f1b7aea7b"
server_key_version: "encryptionkey@0"
I20260812 01:12:49.276610 6800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:12:49.280257 6832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:49.281782 6800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:12:49.281965 6800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "38b10e813657450087b33d374651ffcc"
format_stamp: "Formatted at 2026-08-12 01:12:49 on dist-test-slave-21b9"
server_key: "21d27da2093be3c9e67fa507cb554e2f"
server_key_iv: "412bade3011c9d5ed6bd4c1f1b7aea7b"
server_key_version: "encryptionkey@0"
I20260812 01:12:49.282159 6800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:49.310060 6800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:49.314416 6800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:49.314707 6800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:49.333539 6800 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:45737
I20260812 01:12:49.333592 6911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:45737 every 8 connection(s)
I20260812 01:12:49.334950 6800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:12:49.338924 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:49.340292 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 6800
I20260812 01:12:49.340543 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:12:49.341390 21084 external_mini_cluster.cc:1428] Setting key 0bf857882311c9e3cc558f2de17f6405
I20260812 01:12:49.352332 6913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc: Bootstrap starting.
I20260812 01:12:49.356124 6913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc: Neither blocks nor log segments found. Creating new log.
Aug 12 01:12:49 dist-test-slave-21b9 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:12:49.357312 6913 log.cc:824] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:49.360719 6913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc: No bootstrap required, opened a new log
I20260812 01:12:49.366091 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:49.343588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.20.151.62:45737)
0812 01:12:49.344447 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:49.344463 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:49.344520 (+ 57us) server_negotiation.cc:407] Connection header received
0812 01:12:49.346857 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:49.346888 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:49.347445 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:49.347879 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:49.349419 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:49.351455 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:49.352861 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:49.353884 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:49.354116 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:49.358618 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:49.358632 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:49.358652 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:49.358698 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:49.361745 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:49.362631 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:49.362638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:49.362645 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:49.362823 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:49.363252 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:49.363255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:49.363257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:49.363879 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:49.364119 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:49.364565 (+ 446us) server_negotiation.cc:299] Negotiation successful
0812 01:12:49.364987 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":327,"threads_started":1}
I20260812 01:12:49.369686 6913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b10e813657450087b33d374651ffcc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45737 } }
I20260812 01:12:49.370071 6913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:49.370177 6913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b10e813657450087b33d374651ffcc, State: Initialized, Role: FOLLOWER
I20260812 01:12:49.370711 6913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b10e813657450087b33d374651ffcc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45737 } }
I20260812 01:12:49.370930 6913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:12:49.371059 6913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:12:49.371217 6913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:49.373481 6913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b10e813657450087b33d374651ffcc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45737 } }
I20260812 01:12:49.373970 6913 leader_election.cc:302] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b10e813657450087b33d374651ffcc; no voters:
I20260812 01:12:49.374569 6913 leader_election.cc:288] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:12:49.374658 6925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:12:49.376713 6913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:12:49.378417 6925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [term 1 LEADER]: Becoming Leader. State: Replica: 38b10e813657450087b33d374651ffcc, State: Running, Role: LEADER
I20260812 01:12:49.379040 6925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b10e813657450087b33d374651ffcc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45737 } }
I20260812 01:12:49.382740 6927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b10e813657450087b33d374651ffcc. Latest consensus state: current_term: 1 leader_uuid: "38b10e813657450087b33d374651ffcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b10e813657450087b33d374651ffcc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45737 } } }
I20260812 01:12:49.383493 6927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [sys.catalog]: This master's current role is: LEADER
I20260812 01:12:49.382834 6926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38b10e813657450087b33d374651ffcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b10e813657450087b33d374651ffcc" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45737 } } }
I20260812 01:12:49.383877 6926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc [sys.catalog]: This master's current role is: LEADER
W20260812 01:12:49.387933 6948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:12:49.388065 6948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:12:49.388204 6951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:12:49.389982 6951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:12:49.398772 6951 catalog_manager.cc:1347] Generated new cluster ID: 4b0de52b3bbe4b618bfcd3ea658285aa
I20260812 01:12:49.398895 6951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:12:49.438355 6951 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:12:49.439775 6951 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:12:49.453281 6951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc: Generated new TSK 0
I20260812 01:12:49.454195 6951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:12:49.534551 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45737
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:49.897230 6958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:49.897565 6958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:49.897688 6958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:49.905454 6958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:49.905592 6958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:49.905735 6958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:12:49.916425 6958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:49.917804 6958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:49.919164 6958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:49.933194 6965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:49.933270 6968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:49.933259 6970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:49.934062 6958 server_base.cc:1034] running on GCE node
I20260812 01:12:49.934765 6958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:49.935964 6958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:49.937219 6958 hybrid_clock.cc:648] HybridClock initialized: now 1786497169937151 us; error 57 us; skew 500 ppm
I20260812 01:12:49.937569 6958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:49 dist-test-slave-21b9 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:49.940941 6958 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:12:49.942467 6958 webserver.cc:460] Webserver started at http://127.20.151.1:42793/ using document root <none> and password file <none>
I20260812 01:12:49.943186 6958 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:49.943351 6958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:49.943662 6958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:49.946774 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "204e2dbba12641b9b4655a7212fd2362"
format_stamp: "Formatted at 2026-08-12 01:12:49 on dist-test-slave-21b9"
server_key: "f3d1332999f21374c1f948b9e06ca329"
server_key_iv: "d5cf5a7ec09bdd0b61562f6b0cb54440"
server_key_version: "encryptionkey@0"
I20260812 01:12:49.947597 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "204e2dbba12641b9b4655a7212fd2362"
format_stamp: "Formatted at 2026-08-12 01:12:49 on dist-test-slave-21b9"
server_key: "f3d1332999f21374c1f948b9e06ca329"
server_key_iv: "d5cf5a7ec09bdd0b61562f6b0cb54440"
server_key_version: "encryptionkey@0"
I20260812 01:12:49.953059 6958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 01:12:49.956861 6981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:49.958310 6958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:12:49.958483 6958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "204e2dbba12641b9b4655a7212fd2362"
format_stamp: "Formatted at 2026-08-12 01:12:49 on dist-test-slave-21b9"
server_key: "f3d1332999f21374c1f948b9e06ca329"
server_key_iv: "d5cf5a7ec09bdd0b61562f6b0cb54440"
server_key_version: "encryptionkey@0"
I20260812 01:12:49.958688 6958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:49.981624 6958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:49.986124 6958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:49.986426 6958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:49.987555 6958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:49.989003 6958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:49.989089 6958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:49.989267 6958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:49.989328 6958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:50.026871 6958 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:45001
I20260812 01:12:50.026933 7180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:45001 every 8 connection(s)
I20260812 01:12:50.028258 6958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:12:50.038157 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 6958
I20260812 01:12:50.038353 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:12:50.039031 21084 external_mini_cluster.cc:1428] Setting key d9fb1903b3d8395eebd36293ca468903
Aug 12 01:12:50 dist-test-slave-21b9 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:12:50.049444 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:50.030987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:48997 (local address 127.20.151.62:45737)
0812 01:12:50.031456 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:50.031461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:50.032531 (+ 1070us) server_negotiation.cc:407] Connection header received
0812 01:12:50.034450 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:50.034454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:50.034552 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:50.034654 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:50.036387 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:50.037149 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:50.039165 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:50.039663 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:50.039788 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:50.043689 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:50.043694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:50.043698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:50.043733 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:50.046211 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:50.046974 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:50.046978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:50.046980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:50.047118 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:50.047601 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:50.047605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:50.047607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:50.047878 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:50.048026 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:50.048779 (+ 753us) server_negotiation.cc:299] Negotiation successful
0812 01:12:50.049043 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":215,"threads_started":1}
I20260812 01:12:50.050388 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:50.031557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:45737 (local address 127.20.151.1:48997)
0812 01:12:50.032338 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:50.032381 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:50.034172 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:50.034925 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:50.034933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:50.035578 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:50.036195 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:50.036214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:50.037321 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:50.037325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:50.038913 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:50.038927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:50.039922 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:50.039931 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:50.040224 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:50.041045 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:50.041078 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:50.043449 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:50.046356 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:50.046367 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:50.046384 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:50.046844 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:50.047227 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:50.047232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:50.047236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:50.047474 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:50.048009 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:50.048017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:50.048379 (+ 362us) client_negotiation.cc:769] Sending connection context
0812 01:12:50.048638 (+ 259us) client_negotiation.cc:240] Negotiation successful
0812 01:12:50.049225 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":311,"threads_started":1}
I20260812 01:12:50.052397 7181 heartbeater.cc:347] Connected to a master server at 127.20.151.62:45737
I20260812 01:12:50.052734 7181 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:50.053535 7181 heartbeater.cc:511] Master 127.20.151.62:45737 requested a full tablet report, sending...
I20260812 01:12:50.055768 6862 ts_manager.cc:194] Registered new tserver with Master: 204e2dbba12641b9b4655a7212fd2362 (127.20.151.1:45001)
I20260812 01:12:50.057235 6862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:48997
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:12:50.110373 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45737
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:50.466215 7191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:50.466497 7191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:50.466600 7191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:50.474023 7191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:50.474140 7191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:50.474260 7191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:12:50.484426 7191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:50.485755 7191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:50.487178 7191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:12:50.500382 7199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:50.500475 7201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:50.500434 7198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:50.501637 7191 server_base.cc:1034] running on GCE node
I20260812 01:12:50.502331 7191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:50.503438 7191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:50.504684 7191 hybrid_clock.cc:648] HybridClock initialized: now 1786497170504622 us; error 44 us; skew 500 ppm
I20260812 01:12:50.505044 7191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:50 dist-test-slave-21b9 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497170, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:50.508322 7191 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:12:50.509876 7191 webserver.cc:460] Webserver started at http://127.20.151.2:40273/ using document root <none> and password file <none>
I20260812 01:12:50.510597 7191 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:50.510720 7191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:50.511040 7191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:50.514211 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "507866ded59e4f3e877b1d2acda4448a"
format_stamp: "Formatted at 2026-08-12 01:12:50 on dist-test-slave-21b9"
server_key: "6bcc5cedbafc6b807dd463c43e4653d7"
server_key_iv: "ed2c3ca144e077f68dca4010703e379f"
server_key_version: "encryptionkey@0"
I20260812 01:12:50.514966 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "507866ded59e4f3e877b1d2acda4448a"
format_stamp: "Formatted at 2026-08-12 01:12:50 on dist-test-slave-21b9"
server_key: "6bcc5cedbafc6b807dd463c43e4653d7"
server_key_iv: "ed2c3ca144e077f68dca4010703e379f"
server_key_version: "encryptionkey@0"
I20260812 01:12:50.520457 7191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 01:12:50.524039 7212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:50.525408 7191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:12:50.525583 7191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "507866ded59e4f3e877b1d2acda4448a"
format_stamp: "Formatted at 2026-08-12 01:12:50 on dist-test-slave-21b9"
server_key: "6bcc5cedbafc6b807dd463c43e4653d7"
server_key_iv: "ed2c3ca144e077f68dca4010703e379f"
server_key_version: "encryptionkey@0"
I20260812 01:12:50.525785 7191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:50.538475 7191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:50.542650 7191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:50.542991 7191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:50.544054 7191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:50.545435 7191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:50.545535 7191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:50.545653 7191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:50.545723 7191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:50.577675 7191 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:40283
I20260812 01:12:50.577754 7416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:40283 every 8 connection(s)
I20260812 01:12:50.579027 7191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:12:50.583969 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 7191
I20260812 01:12:50.584137 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:12:50.584578 21084 external_mini_cluster.cc:1428] Setting key 41e676c790d641aa57fe49ee146c79fd
Aug 12 01:12:50 dist-test-slave-21b9 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497170, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:12:50.601213 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:50.581750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:48207 (local address 127.20.151.62:45737)
0812 01:12:50.582194 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:50.582201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:50.583332 (+ 1131us) server_negotiation.cc:407] Connection header received
0812 01:12:50.585416 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:50.585421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:50.585492 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:50.585596 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:50.587380 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:50.588108 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:50.589656 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:50.590105 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:50.590220 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:50.594505 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:50.594511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:50.594516 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:50.594564 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:50.597485 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:50.598490 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:50.598494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:50.598497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:50.598638 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:50.599175 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:50.599180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:50.599182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:50.599566 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:50.599696 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:50.600644 (+ 948us) server_negotiation.cc:299] Negotiation successful
0812 01:12:50.600867 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":199,"threads_started":1}
I20260812 01:12:50.602725 7427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:50.582357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:45737 (local address 127.20.151.2:48207)
0812 01:12:50.583161 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:50.583202 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:50.585128 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:50.585882 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:50.585890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:50.586599 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:50.587198 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:50.587218 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:50.588262 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:50.588265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:50.589509 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:50.589522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:50.590330 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:50.590338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:50.590578 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:50.591351 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:50.591379 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:50.594214 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:50.597683 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:50.597694 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:50.597713 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:50.598370 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:50.598743 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:50.598748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:50.598751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:50.599048 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:50.599703 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:50.599712 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:50.600224 (+ 512us) client_negotiation.cc:769] Sending connection context
0812 01:12:50.600555 (+ 331us) client_negotiation.cc:240] Negotiation successful
0812 01:12:50.601177 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":317,"threads_started":1}
I20260812 01:12:50.604485 7418 heartbeater.cc:347] Connected to a master server at 127.20.151.62:45737
I20260812 01:12:50.604844 7418 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:50.605698 7418 heartbeater.cc:511] Master 127.20.151.62:45737 requested a full tablet report, sending...
I20260812 01:12:50.607540 6862 ts_manager.cc:194] Registered new tserver with Master: 507866ded59e4f3e877b1d2acda4448a (127.20.151.2:40283)
I20260812 01:12:50.608487 6862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:48207
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:12:50.654821 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45737
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:12:51.016594 7431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:12:51.017030 7431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:12:51.017204 7431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:12:51.028930 7431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:12:51.029081 7431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:12:51.029347 7431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:12:51.044898 7431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:12:51.046653 7431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:12:51.048493 7431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 01:12:51.060354 7181 heartbeater.cc:503] Master 127.20.151.62:45737 was elected leader, sending a full tablet report...
W20260812 01:12:51.065145 7438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:51.065039 7440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:12:51.065344 7446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:12:51.066098 7431 server_base.cc:1034] running on GCE node
I20260812 01:12:51.066907 7431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:12:51.068229 7431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:12:51.069482 7431 hybrid_clock.cc:648] HybridClock initialized: now 1786497171069405 us; error 62 us; skew 500 ppm
I20260812 01:12:51.069891 7431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:12:51 dist-test-slave-21b9 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497171, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:12:51.074226 7431 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:12:51.076222 7431 webserver.cc:460] Webserver started at http://127.20.151.3:35197/ using document root <none> and password file <none>
I20260812 01:12:51.077208 7431 fs_manager.cc:359] Metadata directory not provided
I20260812 01:12:51.077391 7431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:12:51.077800 7431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:12:51.081961 7431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "4b4b5b72bb924af28fad70f42fd073f1"
format_stamp: "Formatted at 2026-08-12 01:12:51 on dist-test-slave-21b9"
server_key: "350fb57d2d15030fe280f8f7aa21cdb7"
server_key_iv: "03574ac6495f01d5002a239554a150de"
server_key_version: "encryptionkey@0"
I20260812 01:12:51.083031 7431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "4b4b5b72bb924af28fad70f42fd073f1"
format_stamp: "Formatted at 2026-08-12 01:12:51 on dist-test-slave-21b9"
server_key: "350fb57d2d15030fe280f8f7aa21cdb7"
server_key_iv: "03574ac6495f01d5002a239554a150de"
server_key_version: "encryptionkey@0"
I20260812 01:12:51.090397 7431 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260812 01:12:51.095217 7456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:51.097007 7431 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 01:12:51.097200 7431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "4b4b5b72bb924af28fad70f42fd073f1"
format_stamp: "Formatted at 2026-08-12 01:12:51 on dist-test-slave-21b9"
server_key: "350fb57d2d15030fe280f8f7aa21cdb7"
server_key_iv: "03574ac6495f01d5002a239554a150de"
server_key_version: "encryptionkey@0"
I20260812 01:12:51.097438 7431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:12:51.128458 7431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:12:51.133001 7431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:12:51.133265 7431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:12:51.134217 7431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:12:51.135638 7431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:12:51.135775 7431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:51.135882 7431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:12:51.135927 7431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:12:51.167016 7431 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:45553
I20260812 01:12:51.167073 7642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:45553 every 8 connection(s)
I20260812 01:12:51.168321 7431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:12:51.169435 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 7431
I20260812 01:12:51.169620 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:12:51.170037 21084 external_mini_cluster.cc:1428] Setting key 1f259f57073f2925c8aad2dd800be79d
Aug 12 01:12:51 dist-test-slave-21b9 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497171, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:12:51.189069 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.171036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:40449 (local address 127.20.151.62:45737)
0812 01:12:51.171556 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:51.171563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:51.172691 (+ 1128us) server_negotiation.cc:407] Connection header received
0812 01:12:51.174476 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:51.174480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:51.174559 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:51.174650 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:51.176156 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.177050 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.178574 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.178971 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.179072 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.182524 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:51.182530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:51.182534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:51.182568 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:51.186067 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:51.186903 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:51.186907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:51.186910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:51.187047 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:51.187481 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:51.187485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:51.187487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:51.187768 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:51.187887 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:51.188604 (+ 717us) server_negotiation.cc:299] Negotiation successful
0812 01:12:51.188781 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":263,"threads_started":1}
I20260812 01:12:51.190076 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.171718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:45737 (local address 127.20.151.3:40449)
0812 01:12:51.172526 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:51.172575 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:51.174233 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:51.174831 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:51.174839 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:51.175424 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:12:51.175996 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.176014 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.177168 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.177172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.178452 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.178469 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.179146 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.179151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.179388 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.179996 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:12:51.180025 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:12:51.182300 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:12:51.186201 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:51.186209 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:51.186223 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:51.186771 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:51.187146 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:12:51.187149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:12:51.187152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:12:51.187388 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:12:51.187869 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:12:51.187876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:12:51.188215 (+ 339us) client_negotiation.cc:769] Sending connection context
0812 01:12:51.188450 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 01:12:51.188867 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":337,"threads_started":1}
I20260812 01:12:51.191588 7643 heartbeater.cc:347] Connected to a master server at 127.20.151.62:45737
I20260812 01:12:51.191890 7643 heartbeater.cc:464] Registering TS with master...
I20260812 01:12:51.192605 7643 heartbeater.cc:511] Master 127.20.151.62:45737 requested a full tablet report, sending...
I20260812 01:12:51.194074 6862 ts_manager.cc:194] Registered new tserver with Master: 4b4b5b72bb924af28fad70f42fd073f1 (127.20.151.3:45553)
I20260812 01:12:51.194892 6862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:40449
I20260812 01:12:51.198316 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:12:51.216545 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.202827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.20.151.62:45737)
0812 01:12:51.203030 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:51.203037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:51.203579 (+ 542us) server_negotiation.cc:407] Connection header received
0812 01:12:51.203765 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:51.203771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:51.203849 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:51.203944 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:51.205490 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.206284 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.207654 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.208091 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.208215 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.210722 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:51.210728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:51.210731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:51.210768 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:51.213541 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:51.214282 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:51.214288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:51.214293 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:51.214459 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:51.214866 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:51.214871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:51.214873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:51.215236 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:51.215530 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:51.215907 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 01:12:51.216153 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 01:12:51.220959 6862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:12:51.227880 6862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:12:51.253304 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.241077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.20.151.3:45553)
0812 01:12:51.241807 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:51.241817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:51.242889 (+ 1072us) server_negotiation.cc:407] Connection header received
0812 01:12:51.243174 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:51.243182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:51.243477 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:51.243685 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:51.244812 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.246598 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.250125 (+ 3527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.251609 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.251743 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.252003 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:51.252754 (+ 751us) server_negotiation.cc:299] Negotiation successful
0812 01:12:51.252942 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":383,"threads_started":1}
I20260812 01:12:51.253327 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.240618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:45553 (local address 127.0.0.1:47014)
0812 01:12:51.242713 (+ 2095us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:51.242759 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:51.242946 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:51.244114 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:51.244123 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:51.244160 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:51.244573 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.244587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.246804 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.246810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.249554 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.249573 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.252184 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.252196 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.252452 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.252533 (+ 81us) client_negotiation.cc:769] Sending connection context
0812 01:12:51.252687 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 01:12:51.252890 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1847,"mutex_wait_us":6,"thread_start_us":2453,"threads_started":1}
I20260812 01:12:51.254392 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.240838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:45001 (local address 127.0.0.1:42780)
0812 01:12:51.243372 (+ 2534us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:51.243388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:51.243509 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:51.244437 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:51.244448 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:51.244477 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:51.244947 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.244959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.247065 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.247075 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.249897 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.249908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.252222 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.252229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.252446 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.253791 (+ 1345us) client_negotiation.cc:769] Sending connection context
0812 01:12:51.253884 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 01:12:51.254020 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2400,"mutex_wait_us":1621,"thread_start_us":1495,"threads_started":1}
I20260812 01:12:51.254436 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.241309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.20.151.1:45001)
0812 01:12:51.242365 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:51.242373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:51.243635 (+ 1262us) server_negotiation.cc:407] Connection header received
0812 01:12:51.243755 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:51.243762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:51.243960 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:51.244183 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:51.245195 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.246869 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.250049 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.251600 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.251803 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.252003 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:51.253956 (+ 1953us) server_negotiation.cc:299] Negotiation successful
0812 01:12:51.254123 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":666,"threads_started":1}
I20260812 01:12:51.257848 7070 tablet_service.cc:1467] Processing CreateTablet for tablet c84ddc08aa9740f8a66368e3e647e882 (DEFAULT_TABLE table=test-table [id=0f90d59e9fa44fd0941d340bfe875c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:51.257766 7541 tablet_service.cc:1467] Processing CreateTablet for tablet c84ddc08aa9740f8a66368e3e647e882 (DEFAULT_TABLE table=test-table [id=0f90d59e9fa44fd0941d340bfe875c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:51.259364 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84ddc08aa9740f8a66368e3e647e882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:51.259375 7070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84ddc08aa9740f8a66368e3e647e882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:51.260965 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.243501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:40283 (local address 127.0.0.1:34194)
0812 01:12:51.252367 (+ 8866us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:51.252386 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:51.252513 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:51.253356 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:51.253361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:51.253389 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:51.253817 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.253830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.255982 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.255988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.258000 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.258015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.260134 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.260140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.260306 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.260340 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 01:12:51.260441 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 01:12:51.260588 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8724,"mutex_wait_us":3,"thread_start_us":8541,"threads_started":1}
I20260812 01:12:51.261023 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.243794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.20.151.2:40283)
0812 01:12:51.249783 (+ 5989us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:51.249791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:51.252562 (+ 2771us) server_negotiation.cc:407] Connection header received
0812 01:12:51.252691 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:51.252700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:51.252952 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:51.253167 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:51.254011 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.255812 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.258189 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.259665 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.259803 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.259991 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:51.260518 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 01:12:51.260695 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5839,"thread_start_us":5714,"threads_started":1}
I20260812 01:12:51.265117 7299 tablet_service.cc:1467] Processing CreateTablet for tablet c84ddc08aa9740f8a66368e3e647e882 (DEFAULT_TABLE table=test-table [id=0f90d59e9fa44fd0941d340bfe875c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:12:51.266631 7299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84ddc08aa9740f8a66368e3e647e882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:12:51.272075 7673 tablet_bootstrap.cc:492] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: Bootstrap starting.
I20260812 01:12:51.274096 7674 tablet_bootstrap.cc:492] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362: Bootstrap starting.
I20260812 01:12:51.276687 7673 tablet_bootstrap.cc:654] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:51.277845 7674 tablet_bootstrap.cc:654] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:51.278034 7673 log.cc:824] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:51.279050 7674 log.cc:824] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:51.280570 7678 tablet_bootstrap.cc:492] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: Bootstrap starting.
I20260812 01:12:51.282254 7673 tablet_bootstrap.cc:492] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: No bootstrap required, opened a new log
I20260812 01:12:51.283077 7673 ts_tablet_manager.cc:1397] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260812 01:12:51.283819 7678 tablet_bootstrap.cc:654] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: Neither blocks nor log segments found. Creating new log.
I20260812 01:12:51.284754 7678 log.cc:824] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: Log is configured to *not* fsync() on all Append() calls
I20260812 01:12:51.287475 7678 tablet_bootstrap.cc:492] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: No bootstrap required, opened a new log
I20260812 01:12:51.287721 7678 ts_tablet_manager.cc:1397] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260812 01:12:51.293800 7674 tablet_bootstrap.cc:492] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362: No bootstrap required, opened a new log
I20260812 01:12:51.293655 7673 raft_consensus.cc:348] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4b5b72bb924af28fad70f42fd073f1" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45553 } } peers { permanent_uuid: "204e2dbba12641b9b4655a7212fd2362" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45001 } } peers { permanent_uuid: "507866ded59e4f3e877b1d2acda4448a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40283 } }
I20260812 01:12:51.294092 7673 raft_consensus.cc:374] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:51.294132 7674 ts_tablet_manager.cc:1397] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.009s
I20260812 01:12:51.294196 7673 raft_consensus.cc:729] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4b5b72bb924af28fad70f42fd073f1, State: Initialized, Role: FOLLOWER
I20260812 01:12:51.294806 7673 consensus_queue.cc:260] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4b5b72bb924af28fad70f42fd073f1" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45553 } } peers { permanent_uuid: "204e2dbba12641b9b4655a7212fd2362" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45001 } } peers { permanent_uuid: "507866ded59e4f3e877b1d2acda4448a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40283 } }
I20260812 01:12:51.295954 7643 heartbeater.cc:503] Master 127.20.151.62:45737 was elected leader, sending a full tablet report...
I20260812 01:12:51.296628 7673 ts_tablet_manager.cc:1428] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260812 01:12:51.296674 7678 raft_consensus.cc:348] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4b5b72bb924af28fad70f42fd073f1" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45553 } } peers { permanent_uuid: "204e2dbba12641b9b4655a7212fd2362" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45001 } } peers { permanent_uuid: "507866ded59e4f3e877b1d2acda4448a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40283 } }
I20260812 01:12:51.297151 7678 raft_consensus.cc:374] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:51.297240 7678 raft_consensus.cc:729] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 507866ded59e4f3e877b1d2acda4448a, State: Initialized, Role: FOLLOWER
I20260812 01:12:51.297942 7678 consensus_queue.cc:260] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4b5b72bb924af28fad70f42fd073f1" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45553 } } peers { permanent_uuid: "204e2dbba12641b9b4655a7212fd2362" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45001 } } peers { permanent_uuid: "507866ded59e4f3e877b1d2acda4448a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40283 } }
I20260812 01:12:51.299187 7418 heartbeater.cc:503] Master 127.20.151.62:45737 was elected leader, sending a full tablet report...
I20260812 01:12:51.299952 7678 ts_tablet_manager.cc:1428] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260812 01:12:51.304266 7674 raft_consensus.cc:348] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4b5b72bb924af28fad70f42fd073f1" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45553 } } peers { permanent_uuid: "204e2dbba12641b9b4655a7212fd2362" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45001 } } peers { permanent_uuid: "507866ded59e4f3e877b1d2acda4448a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40283 } }
I20260812 01:12:51.304654 7674 raft_consensus.cc:374] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:12:51.304770 7674 raft_consensus.cc:729] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204e2dbba12641b9b4655a7212fd2362, State: Initialized, Role: FOLLOWER
I20260812 01:12:51.305357 7674 consensus_queue.cc:260] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4b5b72bb924af28fad70f42fd073f1" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45553 } } peers { permanent_uuid: "204e2dbba12641b9b4655a7212fd2362" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45001 } } peers { permanent_uuid: "507866ded59e4f3e877b1d2acda4448a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40283 } }
I20260812 01:12:51.306879 7674 ts_tablet_manager.cc:1428] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260812 01:12:51.333781 7421 tablet.cc:2369] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:12:51.398756 7684 raft_consensus.cc:482] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:12:51.399101 7684 raft_consensus.cc:504] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4b5b72bb924af28fad70f42fd073f1" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45553 } } peers { permanent_uuid: "204e2dbba12641b9b4655a7212fd2362" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45001 } } peers { permanent_uuid: "507866ded59e4f3e877b1d2acda4448a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40283 } }
I20260812 01:12:51.401268 7684 leader_election.cc:288] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b4b5b72bb924af28fad70f42fd073f1 (127.20.151.3:45553), 204e2dbba12641b9b4655a7212fd2362 (127.20.151.1:45001)
I20260812 01:12:51.409293 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.401994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:39163 (local address 127.20.151.1:45001)
0812 01:12:51.402187 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:51.402191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:51.403138 (+ 947us) server_negotiation.cc:407] Connection header received
0812 01:12:51.403343 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:51.403349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:51.403417 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:51.403507 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:51.404278 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.404996 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.407456 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.408289 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.408435 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.408518 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:51.408917 (+ 399us) server_negotiation.cc:299] Negotiation successful
0812 01:12:51.409053 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 01:12:51.409303 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.401716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:45521 (local address 127.20.151.3:45553)
0812 01:12:51.401947 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:51.401955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:51.402880 (+ 925us) server_negotiation.cc:407] Connection header received
0812 01:12:51.403286 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:51.403294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:51.403379 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:51.403460 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:12:51.404175 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.405010 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.407463 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.408287 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.408458 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.408546 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:51.408860 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 01:12:51.409016 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 01:12:51.409413 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.401607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:45553 (local address 127.20.151.2:45521)
0812 01:12:51.402755 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:51.402772 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:51.403005 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:51.403497 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:51.403502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:51.403527 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:51.403912 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.403926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.405181 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.405186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.407262 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.407302 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.408533 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.408539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.408697 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.408722 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:12:51.408796 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 01:12:51.408915 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"mutex_wait_us":88,"thread_start_us":747,"threads_started":1}
I20260812 01:12:51.409489 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.401829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:45001 (local address 127.20.151.2:39163)
0812 01:12:51.403002 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0812 01:12:51.403019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:12:51.403176 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:12:51.403542 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:12:51.403547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:12:51.403574 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:12:51.404121 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.404137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.405139 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.405145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.407289 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:12:51.407307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.408583 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.408591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:12:51.408745 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.408770 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:12:51.408873 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 01:12:51.409008 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":80,"thread_start_us":746,"threads_started":1}
I20260812 01:12:51.410923 7570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84ddc08aa9740f8a66368e3e647e882" candidate_uuid: "507866ded59e4f3e877b1d2acda4448a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4b5b72bb924af28fad70f42fd073f1" is_pre_election: true
I20260812 01:12:51.410918 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84ddc08aa9740f8a66368e3e647e882" candidate_uuid: "507866ded59e4f3e877b1d2acda4448a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204e2dbba12641b9b4655a7212fd2362" is_pre_election: true
I20260812 01:12:51.411576 7570 raft_consensus.cc:2454] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 507866ded59e4f3e877b1d2acda4448a in term 0.
I20260812 01:12:51.411576 7098 raft_consensus.cc:2454] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 507866ded59e4f3e877b1d2acda4448a in term 0.
I20260812 01:12:51.412375 7220 leader_election.cc:302] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4b5b72bb924af28fad70f42fd073f1, 507866ded59e4f3e877b1d2acda4448a; no voters:
I20260812 01:12:51.412858 7684 raft_consensus.cc:2781] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:12:51.412998 7684 raft_consensus.cc:482] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:12:51.413100 7684 raft_consensus.cc:3037] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:51.415376 7684 raft_consensus.cc:504] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4b5b72bb924af28fad70f42fd073f1" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45553 } } peers { permanent_uuid: "204e2dbba12641b9b4655a7212fd2362" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45001 } } peers { permanent_uuid: "507866ded59e4f3e877b1d2acda4448a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40283 } }
I20260812 01:12:51.415968 7684 leader_election.cc:288] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [CANDIDATE]: Term 1 election: Requested vote from peers 4b4b5b72bb924af28fad70f42fd073f1 (127.20.151.3:45553), 204e2dbba12641b9b4655a7212fd2362 (127.20.151.1:45001)
I20260812 01:12:51.416433 7570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84ddc08aa9740f8a66368e3e647e882" candidate_uuid: "507866ded59e4f3e877b1d2acda4448a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4b5b72bb924af28fad70f42fd073f1"
I20260812 01:12:51.416456 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84ddc08aa9740f8a66368e3e647e882" candidate_uuid: "507866ded59e4f3e877b1d2acda4448a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204e2dbba12641b9b4655a7212fd2362"
I20260812 01:12:51.416625 7570 raft_consensus.cc:3037] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:51.416639 7098 raft_consensus.cc:3037] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:12:51.418828 7098 raft_consensus.cc:2454] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 507866ded59e4f3e877b1d2acda4448a in term 1.
I20260812 01:12:51.418864 7570 raft_consensus.cc:2454] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 507866ded59e4f3e877b1d2acda4448a in term 1.
I20260812 01:12:51.419258 7220 leader_election.cc:302] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 204e2dbba12641b9b4655a7212fd2362, 4b4b5b72bb924af28fad70f42fd073f1, 507866ded59e4f3e877b1d2acda4448a; no voters:
I20260812 01:12:51.419720 7684 raft_consensus.cc:2781] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 1 FOLLOWER]: Leader election won for term 1
W20260812 01:12:51.420056 7645 tablet.cc:2369] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:12:51.420253 7684 raft_consensus.cc:686] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 1 LEADER]: Becoming Leader. State: Replica: 507866ded59e4f3e877b1d2acda4448a, State: Running, Role: LEADER
I20260812 01:12:51.420785 7684 consensus_queue.cc:237] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4b5b72bb924af28fad70f42fd073f1" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45553 } } peers { permanent_uuid: "204e2dbba12641b9b4655a7212fd2362" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45001 } } peers { permanent_uuid: "507866ded59e4f3e877b1d2acda4448a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40283 } }
I20260812 01:12:51.425302 6862 catalog_manager.cc:5495] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a reported cstate change: term changed from 0 to 1, leader changed from <none> to 507866ded59e4f3e877b1d2acda4448a (127.20.151.2). New cstate: current_term: 1 leader_uuid: "507866ded59e4f3e877b1d2acda4448a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4b5b72bb924af28fad70f42fd073f1" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "204e2dbba12641b9b4655a7212fd2362" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "507866ded59e4f3e877b1d2acda4448a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40283 } health_report { overall_health: HEALTHY } } }
I20260812 01:12:51.456931 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.451553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.20.151.2:40283)
0812 01:12:51.451771 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:51.451777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:51.451856 (+ 79us) server_negotiation.cc:407] Connection header received
0812 01:12:51.452097 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:51.452101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:51.452187 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:51.452328 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:12:51.452721 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.453475 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.454832 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.455252 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.455395 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.455951 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:12:51.456467 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:12:51.456595 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:51.456635 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 01:12:51.456709 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:12:51.465658 7570 raft_consensus.cc:1261] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1 [term 1 FOLLOWER]: Refusing update from remote peer 507866ded59e4f3e877b1d2acda4448a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:51.465678 7098 raft_consensus.cc:1261] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362 [term 1 FOLLOWER]: Refusing update from remote peer 507866ded59e4f3e877b1d2acda4448a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:12:51.466610 7691 consensus_queue.cc:1035] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b4b5b72bb924af28fad70f42fd073f1" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:51.467005 7684 consensus_queue.cc:1035] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [LEADER]: Connected to new peer: Peer: permanent_uuid: "204e2dbba12641b9b4655a7212fd2362" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:12:51.493454 6860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:51.493816 6860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:12:51.498140 6860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38b10e813657450087b33d374651ffcc: Sending DeleteTablet for 3 replicas of tablet c84ddc08aa9740f8a66368e3e647e882
I20260812 01:12:51.499433 7070 tablet_service.cc:1514] Processing DeleteTablet for tablet c84ddc08aa9740f8a66368e3e647e882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:51 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:42780
I20260812 01:12:51.499571 7299 tablet_service.cc:1514] Processing DeleteTablet for tablet c84ddc08aa9740f8a66368e3e647e882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:51 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:34194
I20260812 01:12:51.499766 7541 tablet_service.cc:1514] Processing DeleteTablet for tablet c84ddc08aa9740f8a66368e3e647e882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:12:51 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:47014
I20260812 01:12:51.501760 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 6958
I20260812 01:12:51.528723 7710 tablet_replica.cc:330] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362: stopping tablet replica
I20260812 01:12:51.530066 7713 tablet_replica.cc:330] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: stopping tablet replica
I20260812 01:12:51.530339 7710 raft_consensus.cc:2229] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:51.530654 7713 raft_consensus.cc:2229] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:12:51.531100 7710 raft_consensus.cc:2258] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:51.531214 7713 raft_consensus.cc:2258] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:51.531806 7711 tablet_replica.cc:330] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: stopping tablet replica
I20260812 01:12:51.532335 7711 raft_consensus.cc:2229] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:12:51.533237 7711 raft_consensus.cc:2258] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:12:51.533819 7710 ts_tablet_manager.cc:1905] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:51.535382 7711 ts_tablet_manager.cc:1905] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:12:51.537271 7713 ts_tablet_manager.cc:1905] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 01:12:51 dist-test-slave-21b9 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:12:51.541240 7710 ts_tablet_manager.cc:1918] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:51.541476 7710 log.cc:1196] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/c84ddc08aa9740f8a66368e3e647e882
I20260812 01:12:51.542104 7710 ts_tablet_manager.cc:1939] T c84ddc08aa9740f8a66368e3e647e882 P 204e2dbba12641b9b4655a7212fd2362: Deleting consensus metadata
I20260812 01:12:51.544342 6836 catalog_manager.cc:4841] TS 204e2dbba12641b9b4655a7212fd2362 (127.20.151.1:45001): tablet c84ddc08aa9740f8a66368e3e647e882 (table test-table [id=0f90d59e9fa44fd0941d340bfe875c5d]) successfully deleted
I20260812 01:12:51.547683 7711 ts_tablet_manager.cc:1918] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:51.547946 7711 log.cc:1196] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/c84ddc08aa9740f8a66368e3e647e882
I20260812 01:12:51.548360 7713 ts_tablet_manager.cc:1918] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:12:51.548547 7711 ts_tablet_manager.cc:1939] T c84ddc08aa9740f8a66368e3e647e882 P 507866ded59e4f3e877b1d2acda4448a: Deleting consensus metadata
I20260812 01:12:51.548588 7713 log.cc:1196] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/c84ddc08aa9740f8a66368e3e647e882
I20260812 01:12:51.548812 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.503477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.20.151.1:45001)
0812 01:12:51.503689 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:51.503695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:51.525233 (+ 21538us) server_negotiation.cc:407] Connection header received
0812 01:12:51.525883 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:51.525891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:51.526011 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:51.526095 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:51.529481 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.530682 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.532711 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.533309 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.533455 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.539442 (+ 5987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:51.539462 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:51.539479 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:51.539537 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:51.544929 (+ 5392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:51.545935 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:51.545944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:51.545948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:51.546112 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:51.546572 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:51.546577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:51.546581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:51.547041 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:51.547400 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:51.548098 (+ 698us) server_negotiation.cc:299] Negotiation successful
0812 01:12:51.548340 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 01:12:51.549132 7713 ts_tablet_manager.cc:1939] T c84ddc08aa9740f8a66368e3e647e882 P 4b4b5b72bb924af28fad70f42fd073f1: Deleting consensus metadata
I20260812 01:12:51.550634 7176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:51.550712 6842 catalog_manager.cc:4841] TS 507866ded59e4f3e877b1d2acda4448a (127.20.151.2:40283): tablet c84ddc08aa9740f8a66368e3e647e882 (table test-table [id=0f90d59e9fa44fd0941d340bfe875c5d]) successfully deleted
I20260812 01:12:51.551003 6842 catalog_manager.cc:4841] TS 4b4b5b72bb924af28fad70f42fd073f1 (127.20.151.3:45553): tablet c84ddc08aa9740f8a66368e3e647e882 (table test-table [id=0f90d59e9fa44fd0941d340bfe875c5d]) successfully deleted
I20260812 01:12:51.660511 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 6958
I20260812 01:12:51.692864 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 7191
Aug 12 01:12:51 dist-test-slave-21b9 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:12:51.710191 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.693637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.20.151.2:40283)
0812 01:12:51.693809 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:51.693814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:51.693834 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:12:51.694026 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:51.694030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:51.694126 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:51.694205 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:51.695873 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.696726 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.698367 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.698815 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.698908 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.703932 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:51.703947 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:51.703963 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:51.704005 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:51.707166 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:51.708009 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:51.708016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:51.708022 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:51.708195 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:51.708588 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:51.708592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:51.708594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:51.708976 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:51.709349 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:51.709626 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 01:12:51.709858 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:12:51.710870 7411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:51.828269 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 7191
I20260812 01:12:51.855937 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 7431
Aug 12 01:12:51 dist-test-slave-21b9 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:12:51.870736 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:12:51.856857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.20.151.3:45553)
0812 01:12:51.857023 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 01:12:51.857028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:12:51.857047 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:12:51.857229 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:12:51.857234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:12:51.857315 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:12:51.857407 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:12:51.858652 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.859435 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.860854 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:12:51.861251 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:12:51.861340 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:12:51.865494 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:12:51.865510 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:12:51.865527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:12:51.865561 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:12:51.868267 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:51.868974 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:51.868981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:51.868984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:51.869138 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:12:51.869427 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:12:51.869430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:12:51.869433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:12:51.869794 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:12:51.870018 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 01:12:51.870279 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 01:12:51.870450 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 01:12:51.871423 7637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:51.961364 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 7431
I20260812 01:12:51.984944 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 6800
I20260812 01:12:51.986128 6906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:12:52.101590 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 6800
2026-08-12T01:12:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4864 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:12:52 dist-test-slave-21b9 krb5kdc[7727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:12:52 dist-test-slave-21b9 krb5kdc[7727](info): set up 2 sockets
Aug 12 01:12:52 dist-test-slave-21b9 krb5kdc[7727](info): commencing operation
krb5kdc: starting...
W20260812 01:12:53.927690 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.772s user 0.009s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:12:54 dist-test-slave-21b9 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:12:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/rangeradmin_127.20.151.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/rangeradmin_127.20.151.20@KRBTEST.COM.keytab.
W20260812 01:12:54.479323 21084 mini_kdc.cc:262] Time spent creating service keytab for rangeradmin/127.20.151.20@KRBTEST.COM: real 0.107s user 0.000s sys 0.043s
WARNING: no policy specified for rangerlookup/127.20.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/rangerlookup_127.20.151.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/rangerlookup_127.20.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/HTTP_127.20.151.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/HTTP_127.20.151.20@KRBTEST.COM.keytab.
I20260812 01:12:54.632252 21084 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWp05DU/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/postgres -l logfile start
2026-08-12 01:12:57.009 UTC [7778] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 01:12:57.009 UTC [7778] LOG: listening on IPv4 address "127.20.151.20", port 56005
2026-08-12 01:12:57.012 UTC [7778] LOG: listening on Unix socket "/tmp/.s.PGSQL.56005"
2026-08-12 01:12:57.036 UTC [7782] LOG: database system was shut down at 2026-08-12 01:12:55 UTC
2026-08-12 01:12:57.039 UTC [7778] LOG: database system is ready to accept connections
I20260812 01:12:58.178766 21084 mini_postgres.cc:96] Postgres bound to 56005
2026-08-12 01:12:58.206 UTC [7797] FATAL: database "slave" does not exist
127.20.151.20:56005 - accepting connections
I20260812 01:12:58.206935 21084 mini_ranger.cc:161] Starting Ranger...
I20260812 01:12:58.242980 21084 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 01:12:58.715631 21084 mini_ranger.cc:87] Created ranger Postgres database
I20260812 01:12:58.715755 21084 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-admin
2026-08-12 01:12:59,083 [I] DB FLAVOR :POSTGRES
2026-08-12 01:12:59,084 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 01:12:59,084 [I] Checking connection..
2026-08-12 01:12:59,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 01:12:59,356 [I] Checking connection passed.
2026-08-12 01:12:59,356 [I] --------- Verifying version history table ---------
2026-08-12 01:12:59,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:12:59,607 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 01:12:59,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:12:59,879 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 01:12:59,879 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 01:12:59,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 01:13:00.129 UTC [7905] WARNING: there is no transaction in progress
2026-08-12 01:13:00,136 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 01:13:00,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:13:00,408 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 01:13:00,408 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 01:13:00,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 01:13:00,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 01:13:00,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-21b9.c.gcp-upstream.internal','N') ;"
2026-08-12 01:13:01,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 01:13:01,459 [I] Table x_portal_user does not exist in database ranger
2026-08-12 01:13:01,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 01:13:01,731 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 01:13:01,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 01:13:01,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 01:13:02,251 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 01:13:02,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 01:13:03.221 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.237 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.254 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.269 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.285 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.303 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.362 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.369 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.383 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.393 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.403 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.416 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.427 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.433 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.440 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:03.446 UTC [8101] WARNING: there is no transaction in progress
2026-08-12 01:13:04,102 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 01:13:04,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 01:13:04,350 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 01:13:04,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 01:13:04,610 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 01:13:04,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 01:13:04,884 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 01:13:04,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 01:13:05,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 01:13:05,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-21b9.c.gcp-upstream.internal';"
2026-08-12 01:13:05,641 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 01:13:05,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 01:13:05,905 [I] DB_PATCHES have already been applied
I20260812 01:13:05.912287 21084 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWp05DU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 01:13:05.912395 21084 mini_ranger.cc:193] Using host: 127.20.151.20
I20260812 01:13:05.937897 21084 mini_ranger.cc:227] Ranger admin URL: http://127.20.151.20:36679
Aug 12, 2026 1:13:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 1:13:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 1:13:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 1:13:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 1:13:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 1:13:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 1:13:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 01:13:07.232023 21084 mini_ranger.cc:160] Time spent starting Ranger: real 9.025s user 0.000s sys 0.084s
Aug 12, 2026 1:13:28 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 1:13:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 12, 2026 1:13:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 1:13:28 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 1:13:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 01:13:29.553828 21084 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/rangerkms_127.20.151.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/rangerkms_127.20.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.151.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.151.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.151.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/HTTP_127.20.151.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.151.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/HTTP_127.20.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 01:13:29 dist-test-slave-21b9 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497209, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 01:13:29.805227 21084 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 01:13:29.837107 21084 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 01:13:30.739084 21084 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 01:13:30.739260 21084 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-kms
I20260812 01:13:30.739419 21084 mini_ranger_kms.cc:227] Using postgres at 127.20.151.20:56005
2026-08-12 01:13:31,656 [I] DB FLAVOR :POSTGRES
2026-08-12 01:13:31,656 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 01:13:31,656 [I] Checking connection
2026-08-12 01:13:31,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 01:13:31,911 [I] connection success
2026-08-12 01:13:31,912 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 01:13:31,912 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 01:13:31,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 01:13:32,162 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 01:13:32,163 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 01:13:32,163 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 01:13:32,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:56005/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 01:13:32,423 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 01:13:32.972102 21084 mini_ranger_kms.cc:310] Created kms service
I20260812 01:13:33.210253 21084 mini_ranger_kms.cc:326] Created kudu user
I20260812 01:13:33.261387 21084 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 01:13:33 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497209, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:13:33.718667 21084 mini_ranger_kms.cc:384] Added ranger policy
I20260812 01:13:33.720117 21084 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 01:13:33.720183 21084 mini_ranger_kms.cc:240] Using host: 127.20.151.20
I20260812 01:13:33.758941 21084 mini_ranger_kms.cc:276] Ranger KMS PID: 8433
I20260812 01:13:33.759135 21084 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.151.20:37041
Aug 12, 2026 1:13:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 1:13:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 1:13:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 1:13:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 1:13:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 1:13:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 01:13:36 dist-test-slave-21b9 krb5kdc[7727](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786497216, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.151.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:13:36.445315 21084 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.640s user 0.154s sys 0.450s
I20260812 01:13:36.445669 21084 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 01:13:36.445899 21084 mini_ranger_kms.cc:401] 127.20.151.20:37041/kms/v1/keys
Aug 12 01:13:36 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497216, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.151.20@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
Aug 12 01:13:37 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497216, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.151.20@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
Aug 12 01:13:39 dist-test-slave-21b9 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:13:39.723426 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:35445
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:35445
--ranger_config_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:37041/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:13:40.218014 8520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:13:40.218357 8520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:13:40.218555 8520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:13:40.226783 8520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:13:40.226987 8520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:13:40.227129 8520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:13:40.227226 8520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:13:40.227310 8520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:13:40.239918 8520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.151.20:37041/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:35445
--ranger_config_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:35445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:13:40.241467 8520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:13:40.242969 8520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:13:40.257483 8526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:13:40.257455 8528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:13:40.257822 8531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:13:40.259341 8520 server_base.cc:1034] running on GCE node
I20260812 01:13:40.260186 8520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:13:40.261656 8520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:13:40.262929 8520 hybrid_clock.cc:648] HybridClock initialized: now 1786497220262823 us; error 32 us; skew 500 ppm
I20260812 01:13:40.263327 8520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:13:40 dist-test-slave-21b9 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497220, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:13:40.271740 8520 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:13:40.273695 8520 webserver.cc:460] Webserver started at http://127.20.151.62:35063/ using document root <none> and password file <none>
I20260812 01:13:40.274499 8520 fs_manager.cc:359] Metadata directory not provided
I20260812 01:13:40.274628 8520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:13:40.274977 8520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:13:40 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497220, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:13:40.341042 8520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "e5cd249d116c4437ac180fd2ca0045d0"
format_stamp: "Formatted at 2026-08-12 01:13:40 on dist-test-slave-21b9"
server_key: "69df61bfbe6a6c0be6443b022476a49e"
server_key_iv: "a84055079477d398992eb41771a6b651"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:40.342206 8520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "e5cd249d116c4437ac180fd2ca0045d0"
format_stamp: "Formatted at 2026-08-12 01:13:40 on dist-test-slave-21b9"
server_key: "69df61bfbe6a6c0be6443b022476a49e"
server_key_iv: "a84055079477d398992eb41771a6b651"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:40.349077 8520 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260812 01:13:40.372014 8553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:13:40.374042 8520 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 01:13:40.374318 8520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "e5cd249d116c4437ac180fd2ca0045d0"
format_stamp: "Formatted at 2026-08-12 01:13:40 on dist-test-slave-21b9"
server_key: "69df61bfbe6a6c0be6443b022476a49e"
server_key_iv: "a84055079477d398992eb41771a6b651"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:40.374625 8520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:13:40.403609 8520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:13:40.409972 8520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:13:40.410554 8520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:13:40.444483 8520 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:35445
I20260812 01:13:40.444607 8639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:35445 every 8 connection(s)
I20260812 01:13:40.446171 8520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:13:40.447142 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 8520
I20260812 01:13:40.447359 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:13:40.448274 21084 external_mini_cluster.cc:1428] Setting key 43f54b9594404621cc6e11280e5c8eb4
I20260812 01:13:40.452584 8640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 01:13:40 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:13:40.466674 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0: Bootstrap starting.
I20260812 01:13:40.471494 8640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0: Neither blocks nor log segments found. Creating new log.
I20260812 01:13:40.472805 8640 log.cc:824] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0: Log is configured to *not* fsync() on all Append() calls
I20260812 01:13:40.476226 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:40.450882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.20.151.62:35445)
0812 01:13:40.452360 (+ 1478us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:40.452375 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:40.452426 (+ 51us) server_negotiation.cc:407] Connection header received
0812 01:13:40.455313 (+ 2887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:40.455346 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:40.455930 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:40.456435 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:13:40.457657 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:40.459986 (+ 2329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:40.461349 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:40.462376 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:40.462570 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:40.466688 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:13:40.466700 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:13:40.466721 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:13:40.466778 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:13:40.471484 (+ 4706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:40.472249 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:40.472257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:40.472265 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:40.472511 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:40.472876 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:40.472881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:40.472885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:40.473678 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:13:40.473966 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:40.474421 (+ 455us) server_negotiation.cc:299] Negotiation successful
0812 01:13:40.474763 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":886,"threads_started":1}
I20260812 01:13:40.476856 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0: No bootstrap required, opened a new log
I20260812 01:13:40.485087 8640 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd249d116c4437ac180fd2ca0045d0" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35445 } }
I20260812 01:13:40.485471 8640 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:13:40.485565 8640 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5cd249d116c4437ac180fd2ca0045d0, State: Initialized, Role: FOLLOWER
I20260812 01:13:40.486120 8640 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd249d116c4437ac180fd2ca0045d0" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35445 } }
I20260812 01:13:40.486346 8640 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:13:40.486483 8640 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:13:40.486651 8640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:13:40.488957 8640 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd249d116c4437ac180fd2ca0045d0" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35445 } }
I20260812 01:13:40.489478 8640 leader_election.cc:302] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5cd249d116c4437ac180fd2ca0045d0; no voters:
I20260812 01:13:40.490226 8640 leader_election.cc:288] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:13:40.490363 8651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:13:40.491199 8651 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [term 1 LEADER]: Becoming Leader. State: Replica: e5cd249d116c4437ac180fd2ca0045d0, State: Running, Role: LEADER
I20260812 01:13:40.491822 8651 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd249d116c4437ac180fd2ca0045d0" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35445 } }
I20260812 01:13:40.492511 8640 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:13:40.494450 8653 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5cd249d116c4437ac180fd2ca0045d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd249d116c4437ac180fd2ca0045d0" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35445 } } }
I20260812 01:13:40.494886 8653 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [sys.catalog]: This master's current role is: LEADER
I20260812 01:13:40.500291 8655 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5cd249d116c4437ac180fd2ca0045d0. Latest consensus state: current_term: 1 leader_uuid: "e5cd249d116c4437ac180fd2ca0045d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cd249d116c4437ac180fd2ca0045d0" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 35445 } } }
I20260812 01:13:40.500617 8655 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0 [sys.catalog]: This master's current role is: LEADER
I20260812 01:13:40.504375 8666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:13:40.505019 8640 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 01:13:40.520170 8666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:13:40.534372 8666 catalog_manager.cc:1347] Generated new cluster ID: ce6382f4251a40fd879f7f374e52f6d0
I20260812 01:13:40.534572 8666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:13:40.567885 8666 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:13:40.569973 8666 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:13:40.589579 8666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0: Generated new TSK 0
I20260812 01:13:40.590915 8666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:13:41.360208 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:37041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35445
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:13:41 dist-test-slave-21b9 krb5kdc[7727](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786497221, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 01:13:41.883315 8725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:13:41.883769 8725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:13:41.883998 8725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:13:41.896839 8725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:13:41.897006 8725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:13:41.897202 8725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:13:41.913661 8725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.151.20:37041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:13:41.915619 8725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:13:41.917420 8725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:13:41.935913 8738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:13:41.935950 8740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:13:41.942672 8725 server_base.cc:1034] running on GCE node
W20260812 01:13:41.943167 8742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:13:41.944118 8725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:13:41.946069 8725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:13:41.947413 8725 hybrid_clock.cc:648] HybridClock initialized: now 1786497221947273 us; error 86 us; skew 500 ppm
I20260812 01:13:41.947829 8725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:13:41 dist-test-slave-21b9 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497221, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:13:41.952481 8725 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:13:41.955230 8725 webserver.cc:460] Webserver started at http://127.20.151.1:45535/ using document root <none> and password file <none>
I20260812 01:13:41.959424 8725 fs_manager.cc:359] Metadata directory not provided
I20260812 01:13:41.959648 8725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:13:41.960131 8725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:13:41 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497221, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:13:41.981642 8725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "e5442b95b74e482d95eaae4b99360b46"
format_stamp: "Formatted at 2026-08-12 01:13:41 on dist-test-slave-21b9"
server_key: "2cc87319f376b36afa39863ccb0c63ce"
server_key_iv: "4638647be801ebc3216033d4d76d7821"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:41.982946 8725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "e5442b95b74e482d95eaae4b99360b46"
format_stamp: "Formatted at 2026-08-12 01:13:41 on dist-test-slave-21b9"
server_key: "2cc87319f376b36afa39863ccb0c63ce"
server_key_iv: "4638647be801ebc3216033d4d76d7821"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:41.990237 8725 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s
I20260812 01:13:42.016103 8757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:13:42.017812 8725 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.001s
I20260812 01:13:42.018078 8725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "e5442b95b74e482d95eaae4b99360b46"
format_stamp: "Formatted at 2026-08-12 01:13:41 on dist-test-slave-21b9"
server_key: "2cc87319f376b36afa39863ccb0c63ce"
server_key_iv: "4638647be801ebc3216033d4d76d7821"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:42.018333 8725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:13:42.065460 8725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:13:42.070808 8725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:13:42.071223 8725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:13:42.074335 8725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:13:42.088533 8725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:13:42.088825 8725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 01:13:42.089099 8725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:13:42.089304 8725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 01:13:42 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497221, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:13:42.340503 8725 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:37189
I20260812 01:13:42.340577 8953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:37189 every 8 connection(s)
I20260812 01:13:42.342842 8725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:13:42.351769 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 8725
I20260812 01:13:42.352035 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:13:42.352856 21084 external_mini_cluster.cc:1428] Setting key 06e25933d95c9940d013ac16e12649e4
Aug 12 01:13:42 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497221, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:13:42.385207 8957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:42.346397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:36671 (local address 127.20.151.62:35445)
0812 01:13:42.362112 (+ 15715us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:42.362120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:42.363639 (+ 1519us) server_negotiation.cc:407] Connection header received
0812 01:13:42.365295 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:42.365301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:42.365436 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:42.365541 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:13:42.367197 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:42.368308 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:42.370127 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:42.371004 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:42.371134 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:42.376814 (+ 5680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:13:42.376822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:13:42.376827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:13:42.376884 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:13:42.380837 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:42.381956 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:42.381963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:42.381966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:42.382202 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:42.382847 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:42.382852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:42.382856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:42.383298 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:13:42.383476 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:42.384470 (+ 994us) server_negotiation.cc:299] Negotiation successful
0812 01:13:42.384744 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15541,"thread_start_us":15690,"threads_started":1}
I20260812 01:13:42.386523 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:42.347413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:35445 (local address 127.20.151.1:36671)
0812 01:13:42.362970 (+ 15557us) negotiation.cc:107] Waiting for socket to connect
0812 01:13:42.363036 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 01:13:42.364989 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:13:42.365855 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:13:42.365865 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:13:42.366367 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:13:42.367010 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:42.367029 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:42.368502 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:42.368507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:42.369977 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:42.369993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:42.371381 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:42.371391 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:42.371820 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:42.372692 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:13:42.372723 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:13:42.376391 (+ 3668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:13:42.381012 (+ 4621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:13:42.381022 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:13:42.381042 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:13:42.381818 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:13:42.382373 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:13:42.382378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:13:42.382381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:13:42.382718 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:13:42.383441 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:13:42.383451 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:13:42.384008 (+ 557us) client_negotiation.cc:769] Sending connection context
0812 01:13:42.384348 (+ 340us) client_negotiation.cc:240] Negotiation successful
0812 01:13:42.385023 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15213,"thread_start_us":14862,"threads_started":1}
I20260812 01:13:42.388581 8955 heartbeater.cc:347] Connected to a master server at 127.20.151.62:35445
I20260812 01:13:42.388983 8955 heartbeater.cc:464] Registering TS with master...
I20260812 01:13:42.390097 8955 heartbeater.cc:511] Master 127.20.151.62:35445 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
I20260812 01:13:42.393009 8584 ts_manager.cc:194] Registered new tserver with Master: e5442b95b74e482d95eaae4b99360b46 (127.20.151.1:37189)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
I20260812 01:13:42.404067 8584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:36671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:13:42.442543 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:37041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35445
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:13:42 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497221, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
W20260812 01:13:43.116719 8963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:13:43.117210 8963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:13:43.117460 8963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:13:43.130801 8963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:13:43.131003 8963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:13:43.131210 8963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:13:43.149863 8963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.151.20:37041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:13:43.151750 8963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:13:43.153597 8963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:13:43.173035 8981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:13:43.175627 8984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:13:43.183496 8963 server_base.cc:1034] running on GCE node
W20260812 01:13:43.184376 8986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:13:43.185355 8963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:13:43.186976 8963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:13:43.188297 8963 hybrid_clock.cc:648] HybridClock initialized: now 1786497223188175 us; error 106 us; skew 500 ppm
I20260812 01:13:43.188777 8963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:13:43 dist-test-slave-21b9 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:13:43.194496 8963 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:13:43.201550 8963 webserver.cc:460] Webserver started at http://127.20.151.2:38217/ using document root <none> and password file <none>
I20260812 01:13:43.202533 8963 fs_manager.cc:359] Metadata directory not provided
I20260812 01:13:43.202764 8963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:13:43.203153 8963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:13:43 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:13:43.230451 8963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "ef9918f285b145f9b1789d6030a8baca"
format_stamp: "Formatted at 2026-08-12 01:13:43 on dist-test-slave-21b9"
server_key: "7b6994f72d820f1c4f1cfea754b66d4e"
server_key_iv: "ddafec477ef8107b29205eb6489d6954"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:43.231959 8963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "ef9918f285b145f9b1789d6030a8baca"
format_stamp: "Formatted at 2026-08-12 01:13:43 on dist-test-slave-21b9"
server_key: "7b6994f72d820f1c4f1cfea754b66d4e"
server_key_iv: "ddafec477ef8107b29205eb6489d6954"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:43.240386 8963 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.002s
I20260812 01:13:43.276021 9001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:13:43.278301 8963 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.008s
I20260812 01:13:43.278640 8963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "ef9918f285b145f9b1789d6030a8baca"
format_stamp: "Formatted at 2026-08-12 01:13:43 on dist-test-slave-21b9"
server_key: "7b6994f72d820f1c4f1cfea754b66d4e"
server_key_iv: "ddafec477ef8107b29205eb6489d6954"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:43.279017 8963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:13:43.347939 8963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:13:43.354313 8963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:13:43.354817 8963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:13:43.362668 8963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:13:43.371837 8963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:13:43.372028 8963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 01:13:43.372305 8963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:13:43.372507 8963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:13:43.409519 8955 heartbeater.cc:503] Master 127.20.151.62:35445 was elected leader, sending a full tablet report...
I20260812 01:13:43.500962 8963 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:40985
I20260812 01:13:43.500972 9206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:40985 every 8 connection(s)
I20260812 01:13:43.502696 8963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:13:43.504904 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 8963
I20260812 01:13:43.505087 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:13:43.505502 21084 external_mini_cluster.cc:1428] Setting key 5143bedd07a825366536d48d7e9c4764
Aug 12 01:13:43 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:13:43.535893 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:43.508033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:56507 (local address 127.20.151.62:35445)
0812 01:13:43.515799 (+ 7766us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:43.515806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:43.515827 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:13:43.515893 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:43.515898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:43.515987 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:43.516174 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:13:43.518917 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:43.520036 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:43.522346 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:43.522856 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:43.522977 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:43.527775 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:13:43.527782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:13:43.527786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:13:43.527828 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:13:43.531989 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:43.532939 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:43.532944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:43.532947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:43.533167 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:43.533734 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:43.533740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:43.533743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:43.534116 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:13:43.534263 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:43.535224 (+ 961us) server_negotiation.cc:299] Negotiation successful
0812 01:13:43.535532 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7629,"thread_start_us":7482,"threads_started":1}
I20260812 01:13:43.537266 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:43.507336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:35445 (local address 127.20.151.2:56507)
0812 01:13:43.508688 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0812 01:13:43.508746 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 01:13:43.512078 (+ 3332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:13:43.516592 (+ 4514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:13:43.516607 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:13:43.517459 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:13:43.518684 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:43.518715 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:43.520199 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:43.520205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:43.522154 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:43.522181 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:43.523125 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:43.523134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:43.523552 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:43.524302 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:13:43.524332 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:13:43.527518 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:13:43.532160 (+ 4642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:13:43.532170 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:13:43.532190 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:13:43.532801 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:13:43.533286 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:13:43.533290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:13:43.533293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:13:43.533553 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:13:43.534255 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:13:43.534264 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:13:43.534733 (+ 469us) client_negotiation.cc:769] Sending connection context
0812 01:13:43.535072 (+ 339us) client_negotiation.cc:240] Negotiation successful
0812 01:13:43.535773 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":708,"threads_started":1}
I20260812 01:13:43.539306 9208 heartbeater.cc:347] Connected to a master server at 127.20.151.62:35445
I20260812 01:13:43.539691 9208 heartbeater.cc:464] Registering TS with master...
I20260812 01:13:43.540771 9208 heartbeater.cc:511] Master 127.20.151.62:35445 requested a full tablet report, sending...
I20260812 01:13:43.542779 8584 ts_manager.cc:194] Registered new tserver with Master: ef9918f285b145f9b1789d6030a8baca (127.20.151.2:40985)
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
I20260812 01:13:43.543848 8584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:56507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:13:43.583511 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:37041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35445
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:13:44.083060 9220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:13:44.083431 9220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:13:44.083614 9220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:13:44.093055 9220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:13:44.093190 9220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:13:44.093338 9220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:13:44.105392 9220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.151.20:37041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:35445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:13:44.106853 9220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:13:44.108328 9220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:13:44.125334 9226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:13:44.125319 9229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:13:44.125567 9234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:13:44.127262 9220 server_base.cc:1034] running on GCE node
I20260812 01:13:44.127959 9220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:13:44.129156 9220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:13:44.130393 9220 hybrid_clock.cc:648] HybridClock initialized: now 1786497224130308 us; error 79 us; skew 500 ppm
I20260812 01:13:44.130766 9220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:13:44 dist-test-slave-21b9 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:13:44.135113 9220 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:13:44.137018 9220 webserver.cc:460] Webserver started at http://127.20.151.3:35769/ using document root <none> and password file <none>
I20260812 01:13:44.137949 9220 fs_manager.cc:359] Metadata directory not provided
I20260812 01:13:44.138132 9220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:13:44.138492 9220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:13:44 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:13:44.158675 9220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "830dcbb6097d428298117a536e6c0406"
format_stamp: "Formatted at 2026-08-12 01:13:44 on dist-test-slave-21b9"
server_key: "4a6b169a01c00cde491c5db501f43aad"
server_key_iv: "46afc56b0818e15eb083a822bced9ea8"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:44.159864 9220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "830dcbb6097d428298117a536e6c0406"
format_stamp: "Formatted at 2026-08-12 01:13:44 on dist-test-slave-21b9"
server_key: "4a6b169a01c00cde491c5db501f43aad"
server_key_iv: "46afc56b0818e15eb083a822bced9ea8"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:44.167640 9220 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 01:13:44.188814 9244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:13:44.190688 9220 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.002s
I20260812 01:13:44.190896 9220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "830dcbb6097d428298117a536e6c0406"
format_stamp: "Formatted at 2026-08-12 01:13:44 on dist-test-slave-21b9"
server_key: "4a6b169a01c00cde491c5db501f43aad"
server_key_iv: "46afc56b0818e15eb083a822bced9ea8"
server_key_version: "kuduclusterkey@0"
I20260812 01:13:44.191147 9220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:13:44.210197 9220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:13:44.215619 9220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:13:44.215983 9220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:13:44.217686 9220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:13:44.219844 9220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:13:44.219949 9220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:13:44.220077 9220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:13:44.220139 9220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:13:44.302894 9220 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:40797
I20260812 01:13:44.302976 9456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:40797 every 8 connection(s)
I20260812 01:13:44.304780 9220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:13:44.312832 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 9220
I20260812 01:13:44.313060 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:13:44.313589 21084 external_mini_cluster.cc:1428] Setting key 60413cb02bea26f46336779f2bde1087
Aug 12 01:13:44 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:13:44.331050 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.308017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:40195 (local address 127.20.151.62:35445)
0812 01:13:44.308859 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:44.308863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:44.310150 (+ 1287us) server_negotiation.cc:407] Connection header received
0812 01:13:44.312373 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:44.312379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:44.312476 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:44.312570 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:13:44.314501 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.315512 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.318118 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.318693 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.318797 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.323662 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:13:44.323668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:13:44.323671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:13:44.323706 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:13:44.327389 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:44.328256 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:44.328261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:44.328265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:44.328421 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:44.329053 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:44.329058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:44.329060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:44.329474 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:13:44.329634 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:44.330486 (+ 852us) server_negotiation.cc:299] Negotiation successful
0812 01:13:44.330662 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":561,"threads_started":1}
I20260812 01:13:44.333084 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.308965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:35445 (local address 127.20.151.3:40195)
0812 01:13:44.309974 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0812 01:13:44.310020 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 01:13:44.312101 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:13:44.312844 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:13:44.312853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:13:44.313446 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:13:44.314305 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.314331 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.315695 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.315700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.317943 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.317960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.318937 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.318945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.319235 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.319967 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:13:44.319997 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:13:44.323352 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:13:44.327535 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:13:44.327545 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:13:44.327564 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:13:44.328113 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:13:44.328546 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:13:44.328551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:13:44.328554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:13:44.328919 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:13:44.329626 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:13:44.329635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:13:44.330088 (+ 453us) client_negotiation.cc:769] Sending connection context
0812 01:13:44.330411 (+ 323us) client_negotiation.cc:240] Negotiation successful
0812 01:13:44.331023 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":535,"threads_started":1}
I20260812 01:13:44.335918 9457 heartbeater.cc:347] Connected to a master server at 127.20.151.62:35445
I20260812 01:13:44.336251 9457 heartbeater.cc:464] Registering TS with master...
I20260812 01:13:44.337217 9457 heartbeater.cc:511] Master 127.20.151.62:35445 requested a full tablet report, sending...
I20260812 01:13:44.339004 8584 ts_manager.cc:194] Registered new tserver with Master: 830dcbb6097d428298117a536e6c0406 (127.20.151.3:40797)
I20260812 01:13:44.340232 8584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:40195
I20260812 01:13:44.342381 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:13:44.366919 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.349053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.20.151.62:35445)
0812 01:13:44.349277 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:44.349285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:44.350070 (+ 785us) server_negotiation.cc:407] Connection header received
0812 01:13:44.350317 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:44.350321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:44.350400 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:44.350489 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:13:44.352604 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.353416 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.355959 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.356460 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.356604 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.359744 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:13:44.359751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:13:44.359756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:13:44.359801 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:13:44.363494 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:44.364377 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:44.364383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:44.364386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:44.364607 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:44.365017 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:44.365022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:44.365025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:44.365432 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:13:44.365694 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:44.366330 (+ 636us) server_negotiation.cc:299] Negotiation successful
0812 01:13:44.366592 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:13:44.372076 8584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:13:44.385082 8584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:13:44.422096 9478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.401655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:40985 (local address 127.0.0.1:39636)
0812 01:13:44.407836 (+ 6181us) negotiation.cc:107] Waiting for socket to connect
0812 01:13:44.407890 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 01:13:44.408183 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:13:44.410586 (+ 2403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:13:44.410594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:13:44.410628 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:13:44.411137 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.411150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.413482 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.413488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.417825 (+ 4337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.417841 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.420656 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.420668 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.420961 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.421019 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 01:13:44.421444 (+ 425us) client_negotiation.cc:240] Negotiation successful
0812 01:13:44.421670 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5913,"mutex_wait_us":21,"thread_start_us":5738,"threads_started":1}
I20260812 01:13:44.422176 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.402847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.20.151.2:40985)
0812 01:13:44.403854 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:44.403864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:44.408018 (+ 4154us) server_negotiation.cc:407] Connection header received
0812 01:13:44.408448 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:44.408457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:44.408758 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:44.408933 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:13:44.411468 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.413350 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.418063 (+ 4713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.419831 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.419984 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.420193 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:44.421522 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0812 01:13:44.421701 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":699,"threads_started":1}
I20260812 01:13:44.425958 9487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.401791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:40797 (local address 127.0.0.1:43432)
0812 01:13:44.415204 (+ 13413us) negotiation.cc:107] Waiting for socket to connect
0812 01:13:44.415225 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:13:44.415428 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:13:44.416742 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:13:44.416747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:13:44.416777 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:13:44.417387 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.417397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.419548 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.419553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.422328 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.422342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.424745 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.424755 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.424976 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.425013 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 01:13:44.425088 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 01:13:44.425353 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11560,"mutex_wait_us":23,"thread_start_us":14794,"threads_started":1}
I20260812 01:13:44.427345 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.402351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.20.151.3:40797)
0812 01:13:44.415949 (+ 13598us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:44.415956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:44.416044 (+ 88us) server_negotiation.cc:407] Connection header received
0812 01:13:44.416166 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:44.416175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:44.416487 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:44.416943 (+ 456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:13:44.417615 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.419421 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.422522 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.424344 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.426307 (+ 1963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.426568 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:44.426646 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 01:13:44.426829 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13373,"thread_start_us":13220,"threads_started":1}
I20260812 01:13:44.427440 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.409105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.20.151.1:37189)
0812 01:13:44.415008 (+ 5903us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:44.415020 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:44.417019 (+ 1999us) server_negotiation.cc:407] Connection header received
0812 01:13:44.417355 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:44.417363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:44.417635 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:44.417874 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:13:44.418877 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.420626 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.424681 (+ 4055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.425771 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.425904 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.426168 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:44.426766 (+ 598us) server_negotiation.cc:299] Negotiation successful
0812 01:13:44.426930 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5663,"thread_start_us":5608,"threads_started":1}
I20260812 01:13:44.427605 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.410385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:37189 (local address 127.0.0.1:42034)
0812 01:13:44.416906 (+ 6521us) negotiation.cc:107] Waiting for socket to connect
0812 01:13:44.416922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:13:44.417131 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:13:44.417877 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:13:44.417882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:13:44.417910 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:13:44.418680 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.418696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.420829 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.420834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.424484 (+ 3650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.424498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.426306 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.426317 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.426527 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.426564 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 01:13:44.426698 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 01:13:44.426829 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6372,"mutex_wait_us":975,"thread_start_us":2793,"threads_started":1}
I20260812 01:13:44.427542 9091 tablet_service.cc:1467] Processing CreateTablet for tablet 78d8f8a282644cb4ac7c17c77e8c3924 (DEFAULT_TABLE table=test-table [id=fdda191101194d63b0969445ce8f6e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:13:44.430191 9091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d8f8a282644cb4ac7c17c77e8c3924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:13:44.431384 8829 tablet_service.cc:1467] Processing CreateTablet for tablet 78d8f8a282644cb4ac7c17c77e8c3924 (DEFAULT_TABLE table=test-table [id=fdda191101194d63b0969445ce8f6e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:13:44.432852 8829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d8f8a282644cb4ac7c17c77e8c3924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:13:44.432809 9330 tablet_service.cc:1467] Processing CreateTablet for tablet 78d8f8a282644cb4ac7c17c77e8c3924 (DEFAULT_TABLE table=test-table [id=fdda191101194d63b0969445ce8f6e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:13:44.434335 9330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d8f8a282644cb4ac7c17c77e8c3924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:13:44.452750 9488 tablet_bootstrap.cc:492] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: Bootstrap starting.
I20260812 01:13:44.458799 9490 tablet_bootstrap.cc:492] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: Bootstrap starting.
I20260812 01:13:44.462888 9488 tablet_bootstrap.cc:654] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: Neither blocks nor log segments found. Creating new log.
I20260812 01:13:44.463433 9490 tablet_bootstrap.cc:654] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: Neither blocks nor log segments found. Creating new log.
I20260812 01:13:44.464933 9488 log.cc:824] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: Log is configured to *not* fsync() on all Append() calls
I20260812 01:13:44.465161 9490 log.cc:824] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: Log is configured to *not* fsync() on all Append() calls
I20260812 01:13:44.466523 9491 tablet_bootstrap.cc:492] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: Bootstrap starting.
I20260812 01:13:44.470649 9491 tablet_bootstrap.cc:654] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: Neither blocks nor log segments found. Creating new log.
I20260812 01:13:44.472718 9491 log.cc:824] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: Log is configured to *not* fsync() on all Append() calls
I20260812 01:13:44.473506 9490 tablet_bootstrap.cc:492] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: No bootstrap required, opened a new log
I20260812 01:13:44.473801 9488 tablet_bootstrap.cc:492] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: No bootstrap required, opened a new log
I20260812 01:13:44.473868 9490 ts_tablet_manager.cc:1397] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20260812 01:13:44.474216 9488 ts_tablet_manager.cc:1397] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.009s
I20260812 01:13:44.476132 9491 tablet_bootstrap.cc:492] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: No bootstrap required, opened a new log
I20260812 01:13:44.476478 9491 ts_tablet_manager.cc:1397] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260812 01:13:44.485675 9491 raft_consensus.cc:348] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830dcbb6097d428298117a536e6c0406" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40797 } } peers { permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 } } peers { permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 } }
I20260812 01:13:44.486195 9491 raft_consensus.cc:374] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:13:44.486399 9491 raft_consensus.cc:729] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830dcbb6097d428298117a536e6c0406, State: Initialized, Role: FOLLOWER
I20260812 01:13:44.486490 9490 raft_consensus.cc:348] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830dcbb6097d428298117a536e6c0406" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40797 } } peers { permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 } } peers { permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 } }
I20260812 01:13:44.487033 9490 raft_consensus.cc:374] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:13:44.487229 9490 raft_consensus.cc:729] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9918f285b145f9b1789d6030a8baca, State: Initialized, Role: FOLLOWER
I20260812 01:13:44.487282 9491 consensus_queue.cc:260] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830dcbb6097d428298117a536e6c0406" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40797 } } peers { permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 } } peers { permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 } }
I20260812 01:13:44.487547 9488 raft_consensus.cc:348] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830dcbb6097d428298117a536e6c0406" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40797 } } peers { permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 } } peers { permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 } }
I20260812 01:13:44.488066 9488 raft_consensus.cc:374] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:13:44.488271 9488 raft_consensus.cc:729] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5442b95b74e482d95eaae4b99360b46, State: Initialized, Role: FOLLOWER
I20260812 01:13:44.488211 9490 consensus_queue.cc:260] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830dcbb6097d428298117a536e6c0406" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40797 } } peers { permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 } } peers { permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 } }
I20260812 01:13:44.488785 9457 heartbeater.cc:503] Master 127.20.151.62:35445 was elected leader, sending a full tablet report...
I20260812 01:13:44.488937 9488 consensus_queue.cc:260] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830dcbb6097d428298117a536e6c0406" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40797 } } peers { permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 } } peers { permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 } }
I20260812 01:13:44.489634 9491 ts_tablet_manager.cc:1428] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20260812 01:13:44.493224 9490 ts_tablet_manager.cc:1428] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: Time spent starting tablet: real 0.019s user 0.018s sys 0.003s
I20260812 01:13:44.493299 9208 heartbeater.cc:503] Master 127.20.151.62:35445 was elected leader, sending a full tablet report...
I20260812 01:13:44.494429 9488 ts_tablet_manager.cc:1428] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: Time spent starting tablet: real 0.020s user 0.008s sys 0.012s
W20260812 01:13:44.505879 9209 tablet.cc:2369] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:13:44.557580 9460 tablet.cc:2369] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:13:44.603048 8956 tablet.cc:2369] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:13:44.669269 9496 raft_consensus.cc:482] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:13:44.669773 9496 raft_consensus.cc:504] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830dcbb6097d428298117a536e6c0406" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40797 } } peers { permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 } } peers { permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 } }
I20260812 01:13:44.671710 9496 leader_election.cc:288] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef9918f285b145f9b1789d6030a8baca (127.20.151.2:40985), e5442b95b74e482d95eaae4b99360b46 (127.20.151.1:37189)
I20260812 01:13:44.679752 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.672199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:43809 (local address 127.20.151.2:40985)
0812 01:13:44.672530 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:44.672538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:44.672562 (+ 24us) server_negotiation.cc:407] Connection header received
0812 01:13:44.672652 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:44.672658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:44.672742 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:44.672842 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:13:44.673470 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.674783 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.677158 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.678652 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.678824 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.678886 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:44.679378 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 01:13:44.679479 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 01:13:44.679874 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.671849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:40985 (local address 127.20.151.3:43809)
0812 01:13:44.672064 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 01:13:44.672090 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 01:13:44.672251 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:13:44.673076 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:13:44.673081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:13:44.673109 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:13:44.673311 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.673321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.674945 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.674951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.676981 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.677001 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.678819 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.678825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.679182 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.679221 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 01:13:44.679344 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 01:13:44.679496 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 01:13:44.681313 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.672856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:33483 (local address 127.20.151.1:37189)
0812 01:13:44.673068 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:44.673073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:44.673309 (+ 236us) server_negotiation.cc:407] Connection header received
0812 01:13:44.673502 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:44.673506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:44.673566 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:44.673631 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:13:44.674755 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.675993 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.678869 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.680366 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.680608 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.680686 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:44.680972 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 01:13:44.681091 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 01:13:44.681460 9127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d8f8a282644cb4ac7c17c77e8c3924" candidate_uuid: "830dcbb6097d428298117a536e6c0406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef9918f285b145f9b1789d6030a8baca" is_pre_election: true
I20260812 01:13:44.681778 9498 raft_consensus.cc:482] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:13:44.681892 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.672246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:37189 (local address 127.20.151.3:33483)
0812 01:13:44.673206 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0812 01:13:44.673220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 01:13:44.673343 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:13:44.673679 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:13:44.673684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:13:44.673710 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:13:44.674563 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.674579 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.676175 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.676179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.678707 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.678721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.680621 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.680628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.680802 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.680840 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 01:13:44.680946 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 01:13:44.681124 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":625,"threads_started":1}
I20260812 01:13:44.682102 9127 raft_consensus.cc:2454] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830dcbb6097d428298117a536e6c0406 in term 0.
I20260812 01:13:44.682097 9498 raft_consensus.cc:504] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830dcbb6097d428298117a536e6c0406" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40797 } } peers { permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 } } peers { permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 } }
I20260812 01:13:44.683053 9247 leader_election.cc:302] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830dcbb6097d428298117a536e6c0406, ef9918f285b145f9b1789d6030a8baca; no voters:
I20260812 01:13:44.683152 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d8f8a282644cb4ac7c17c77e8c3924" candidate_uuid: "830dcbb6097d428298117a536e6c0406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5442b95b74e482d95eaae4b99360b46" is_pre_election: true
I20260812 01:13:44.683627 9496 raft_consensus.cc:2781] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:13:44.683866 9496 raft_consensus.cc:482] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:13:44.684101 9496 raft_consensus.cc:3037] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:13:44.683938 8872 raft_consensus.cc:2454] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830dcbb6097d428298117a536e6c0406 in term 0.
I20260812 01:13:44.684896 9498 leader_election.cc:288] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830dcbb6097d428298117a536e6c0406 (127.20.151.3:40797), ef9918f285b145f9b1789d6030a8baca (127.20.151.2:40985)
I20260812 01:13:44.688786 9496 raft_consensus.cc:504] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830dcbb6097d428298117a536e6c0406" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40797 } } peers { permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 } } peers { permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 } }
I20260812 01:13:44.689895 9496 leader_election.cc:288] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [CANDIDATE]: Term 1 election: Requested vote from peers ef9918f285b145f9b1789d6030a8baca (127.20.151.2:40985), e5442b95b74e482d95eaae4b99360b46 (127.20.151.1:37189)
I20260812 01:13:44.690048 9127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d8f8a282644cb4ac7c17c77e8c3924" candidate_uuid: "830dcbb6097d428298117a536e6c0406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef9918f285b145f9b1789d6030a8baca"
I20260812 01:13:44.690173 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d8f8a282644cb4ac7c17c77e8c3924" candidate_uuid: "830dcbb6097d428298117a536e6c0406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5442b95b74e482d95eaae4b99360b46"
I20260812 01:13:44.690276 9127 raft_consensus.cc:3037] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:13:44.690387 8872 raft_consensus.cc:3037] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:13:44.694568 8872 raft_consensus.cc:2454] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830dcbb6097d428298117a536e6c0406 in term 1.
I20260812 01:13:44.694765 9127 raft_consensus.cc:2454] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830dcbb6097d428298117a536e6c0406 in term 1.
I20260812 01:13:44.695145 9247 leader_election.cc:302] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830dcbb6097d428298117a536e6c0406, e5442b95b74e482d95eaae4b99360b46; no voters:
I20260812 01:13:44.695727 9496 raft_consensus.cc:2781] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:13:44.696422 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.685360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:48455 (local address 127.20.151.2:40985)
0812 01:13:44.685888 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:44.685893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:44.687229 (+ 1336us) server_negotiation.cc:407] Connection header received
0812 01:13:44.687431 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:44.687436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:44.687512 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:44.687625 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:13:44.688413 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.689335 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.692523 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.694354 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.694513 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.694587 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:44.695962 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0812 01:13:44.696141 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410}
I20260812 01:13:44.696606 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.685299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:40985 (local address 127.20.151.1:48455)
0812 01:13:44.687112 (+ 1813us) negotiation.cc:107] Waiting for socket to connect
0812 01:13:44.687135 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:13:44.687293 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:13:44.687648 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:13:44.687653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:13:44.687678 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:13:44.688261 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.688272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.689522 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.689528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.692358 (+ 2830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.692375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.695482 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.695493 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.695787 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.695823 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 01:13:44.695928 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 01:13:44.696117 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1590,"thread_start_us":1422,"threads_started":1}
I20260812 01:13:44.698252 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.684906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:40797 (local address 127.20.151.1:59341)
0812 01:13:44.686236 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0812 01:13:44.686251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:13:44.686443 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:13:44.686953 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:13:44.686958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:13:44.686987 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:13:44.687896 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.687913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.689567 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.689576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.695156 (+ 5580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:13:44.695173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.697383 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.697390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:13:44.697581 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.697614 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 01:13:44.697704 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 01:13:44.697828 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"thread_start_us":985,"threads_started":1}
I20260812 01:13:44.698449 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.685373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:59341 (local address 127.20.151.3:40797)
0812 01:13:44.685566 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:44.685572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:44.686338 (+ 766us) server_negotiation.cc:407] Connection header received
0812 01:13:44.686607 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:44.686612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:44.686693 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:44.686782 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:13:44.688047 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.689395 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.695369 (+ 5974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.697174 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.697323 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.697394 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:44.697767 (+ 373us) server_negotiation.cc:299] Negotiation successful
0812 01:13:44.697987 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 01:13:44.698619 9127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d8f8a282644cb4ac7c17c77e8c3924" candidate_uuid: "e5442b95b74e482d95eaae4b99360b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef9918f285b145f9b1789d6030a8baca" is_pre_election: true
I20260812 01:13:44.698941 9127 raft_consensus.cc:2379] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5442b95b74e482d95eaae4b99360b46 in current term 1: Already voted for candidate 830dcbb6097d428298117a536e6c0406 in this term.
I20260812 01:13:44.699522 9366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d8f8a282644cb4ac7c17c77e8c3924" candidate_uuid: "e5442b95b74e482d95eaae4b99360b46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830dcbb6097d428298117a536e6c0406" is_pre_election: true
I20260812 01:13:44.702661 9496 raft_consensus.cc:686] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 1 LEADER]: Becoming Leader. State: Replica: 830dcbb6097d428298117a536e6c0406, State: Running, Role: LEADER
I20260812 01:13:44.703202 9496 consensus_queue.cc:237] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830dcbb6097d428298117a536e6c0406" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40797 } } peers { permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 } } peers { permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 } }
I20260812 01:13:44.707862 8761 leader_election.cc:302] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5442b95b74e482d95eaae4b99360b46; no voters: 830dcbb6097d428298117a536e6c0406, ef9918f285b145f9b1789d6030a8baca
I20260812 01:13:44.708443 9498 raft_consensus.cc:2726] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 01:13:44.709084 8584 catalog_manager.cc:5495] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 reported cstate change: term changed from 0 to 1, leader changed from <none> to 830dcbb6097d428298117a536e6c0406 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "830dcbb6097d428298117a536e6c0406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830dcbb6097d428298117a536e6c0406" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 } health_report { overall_health: UNKNOWN } } }
I20260812 01:13:44.795771 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.789222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.20.151.3:40797)
0812 01:13:44.789456 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:44.789461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:44.789480 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:13:44.789730 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:44.789734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:44.789817 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:44.789943 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:13:44.790257 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.791244 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.793124 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.793670 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.793805 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.794388 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:13:44.794909 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:13:44.795138 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:44.795191 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 01:13:44.795296 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 01:13:44.806110 9127 raft_consensus.cc:1261] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca [term 1 FOLLOWER]: Refusing update from remote peer 830dcbb6097d428298117a536e6c0406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:13:44.806228 8872 raft_consensus.cc:1261] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 1 FOLLOWER]: Refusing update from remote peer 830dcbb6097d428298117a536e6c0406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:13:44.807202 9506 consensus_queue.cc:1035] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef9918f285b145f9b1789d6030a8baca" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:13:44.807610 9496 consensus_queue.cc:1035] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5442b95b74e482d95eaae4b99360b46" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:13:44.822976 9513 mvcc.cc:205] Tried to move back new op lower bound from 7317492632793096192 to 7317492632388345856. Current Snapshot: MvccSnapshot[applied={T|T < 7317492632793096192}]
I20260812 01:13:44.825578 9512 mvcc.cc:205] Tried to move back new op lower bound from 7317492632793096192 to 7317492632388345856. Current Snapshot: MvccSnapshot[applied={T|T < 7317492632793096192}]
I20260812 01:13:44.836709 8581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:13:44.837091 8581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:13:44.841614 8581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5cd249d116c4437ac180fd2ca0045d0: Sending DeleteTablet for 3 replicas of tablet 78d8f8a282644cb4ac7c17c77e8c3924
I20260812 01:13:44.843092 9330 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d8f8a282644cb4ac7c17c77e8c3924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:13:44 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:43432
I20260812 01:13:44.843286 8829 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d8f8a282644cb4ac7c17c77e8c3924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:13:44 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:42034
I20260812 01:13:44.843662 9091 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d8f8a282644cb4ac7c17c77e8c3924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:13:44 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:39636
I20260812 01:13:44.844502 9529 tablet_replica.cc:330] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: stopping tablet replica
I20260812 01:13:44.845166 9529 raft_consensus.cc:2229] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:13:44.845714 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 8725
I20260812 01:13:44.845984 9529 raft_consensus.cc:2258] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:13:44.848450 9527 tablet_replica.cc:330] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: stopping tablet replica
I20260812 01:13:44.849030 9527 raft_consensus.cc:2229] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:13:44.849841 9527 raft_consensus.cc:2258] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:13:44.852233 9529 ts_tablet_manager.cc:1905] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:13:44.852967 9527 ts_tablet_manager.cc:1905] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:13:44.856771 9530 tablet_replica.cc:330] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: stopping tablet replica
I20260812 01:13:44.857273 9530 raft_consensus.cc:2229] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:13:44.857825 9530 raft_consensus.cc:2258] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:13:44.860355 9529 ts_tablet_manager.cc:1918] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:13:44.860425 9530 ts_tablet_manager.cc:1905] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:13:44.860556 9529 log.cc:1196] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/78d8f8a282644cb4ac7c17c77e8c3924
I20260812 01:13:44.861116 9529 ts_tablet_manager.cc:1939] T 78d8f8a282644cb4ac7c17c77e8c3924 P e5442b95b74e482d95eaae4b99360b46: Deleting consensus metadata
Aug 12 01:13:44 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:13:44.863186 8556 catalog_manager.cc:4841] TS e5442b95b74e482d95eaae4b99360b46 (127.20.151.1:37189): tablet 78d8f8a282644cb4ac7c17c77e8c3924 (table test-table [id=fdda191101194d63b0969445ce8f6e37]) successfully deleted
I20260812 01:13:44.866899 9527 ts_tablet_manager.cc:1918] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:13:44.867179 9527 log.cc:1196] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/78d8f8a282644cb4ac7c17c77e8c3924
I20260812 01:13:44.867879 9527 ts_tablet_manager.cc:1939] T 78d8f8a282644cb4ac7c17c77e8c3924 P 830dcbb6097d428298117a536e6c0406: Deleting consensus metadata
I20260812 01:13:44.868660 9530 ts_tablet_manager.cc:1918] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:13:44.868891 9530 log.cc:1196] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/78d8f8a282644cb4ac7c17c77e8c3924
I20260812 01:13:44.869696 9530 ts_tablet_manager.cc:1939] T 78d8f8a282644cb4ac7c17c77e8c3924 P ef9918f285b145f9b1789d6030a8baca: Deleting consensus metadata
I20260812 01:13:44.870380 8561 catalog_manager.cc:4841] TS 830dcbb6097d428298117a536e6c0406 (127.20.151.3:40797): tablet 78d8f8a282644cb4ac7c17c77e8c3924 (table test-table [id=fdda191101194d63b0969445ce8f6e37]) successfully deleted
I20260812 01:13:44.871943 8556 catalog_manager.cc:4841] TS ef9918f285b145f9b1789d6030a8baca (127.20.151.2:40985): tablet 78d8f8a282644cb4ac7c17c77e8c3924 (table test-table [id=fdda191101194d63b0969445ce8f6e37]) successfully deleted
I20260812 01:13:44.873512 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:44.846910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.20.151.1:37189)
0812 01:13:44.847120 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:44.847127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:44.850708 (+ 3581us) server_negotiation.cc:407] Connection header received
0812 01:13:44.851012 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:44.851018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:44.851098 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:44.851177 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:13:44.853449 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.854564 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.856480 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:44.857110 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:44.857214 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:44.863346 (+ 6132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:13:44.863360 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:13:44.863378 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:13:44.863424 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:13:44.868754 (+ 5330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:44.870046 (+ 1292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:44.870055 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:44.870059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:44.870535 (+ 476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:44.871047 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:44.871051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:44.871053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:44.871640 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:13:44.871951 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:44.872368 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 01:13:44.872608 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 01:13:44.874240 8943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:13:44.990636 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 8725
I20260812 01:13:45.022075 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 8963
Aug 12 01:13:45 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:13:45.051080 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:45.022897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.20.151.2:40985)
0812 01:13:45.023084 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:45.023091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:45.023349 (+ 258us) server_negotiation.cc:407] Connection header received
0812 01:13:45.023643 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:45.023649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:45.023735 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:45.023808 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:13:45.026247 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:45.028210 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:45.034180 (+ 5970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:45.034882 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:45.035029 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:45.041261 (+ 6232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:13:45.041287 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:13:45.041312 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:13:45.041371 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:13:45.046091 (+ 4720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:45.047324 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:45.047338 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:45.047342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:45.047594 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:45.049249 (+ 1655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:45.049253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:45.049256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:45.049669 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:13:45.049959 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:45.050422 (+ 463us) server_negotiation.cc:299] Negotiation successful
0812 01:13:45.050688 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 01:13:45.051815 9200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:13:45.150388 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 8963
I20260812 01:13:45.177860 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 9220
Aug 12 01:13:45 dist-test-slave-21b9 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:13:45.197489 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:13:45.178785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.20.151.3:40797)
0812 01:13:45.178984 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 01:13:45.178988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:13:45.179050 (+ 62us) server_negotiation.cc:407] Connection header received
0812 01:13:45.179250 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:13:45.179255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:13:45.179393 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:13:45.179492 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:13:45.181307 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:45.182201 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:45.184285 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:13:45.184737 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:13:45.184837 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:13:45.190719 (+ 5882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:13:45.190733 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:13:45.190753 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:13:45.190806 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:13:45.194037 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:45.195013 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:45.195022 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:45.195024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:45.195189 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:13:45.195688 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:13:45.195693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:13:45.195696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:13:45.196305 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:13:45.196588 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 01:13:45.197006 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 01:13:45.197201 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 01:13:45.198220 9446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:13:45.321952 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 9220
I20260812 01:13:45.348651 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 8520
I20260812 01:13:45.349956 8631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:13:45.465843 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 8520
I20260812 01:13:45.498425 21084 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 01:13:45.844355 21084 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 01:13:45.844667 21084 mini_ranger.cc:66] Stopping Ranger...
I20260812 01:13:46.200799 21084 mini_ranger.cc:68] Stopped Ranger
2026-08-12 01:13:46.201 UTC [7778] LOG: received smart shutdown request
2026-08-12 01:13:46.204 UTC [7778] LOG: background worker "logical replication launcher" (PID 7791) exited with exit code 1
2026-08-12 01:13:46.205 UTC [7783] LOG: shutting down
2026-08-12 01:13:46.370 UTC [7778] LOG: database system is shut down
2026-08-12T01:13:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54367 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:13:46 dist-test-slave-21b9 krb5kdc[9557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:13:46 dist-test-slave-21b9 krb5kdc[9557](info): set up 2 sockets
Aug 12 01:13:46 dist-test-slave-21b9 krb5kdc[9557](info): commencing operation
krb5kdc: starting...
W20260812 01:13:47.770367 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.007s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:13:47 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:13:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/rangeradmin_127.20.151.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/rangeradmin_127.20.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/rangerlookup_127.20.151.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/rangerlookup_127.20.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/HTTP_127.20.151.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/HTTP_127.20.151.20@KRBTEST.COM.keytab.
I20260812 01:13:48.226380 21084 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWp05DU/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/postgres -l logfile start
2026-08-12 01:13:50.819 UTC [9603] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 01:13:50.819 UTC [9603] LOG: listening on IPv4 address "127.20.151.20", port 51097
2026-08-12 01:13:50.823 UTC [9603] LOG: listening on Unix socket "/tmp/.s.PGSQL.51097"
2026-08-12 01:13:50.843 UTC [9606] LOG: database system was shut down at 2026-08-12 01:13:49 UTC
2026-08-12 01:13:50.847 UTC [9603] LOG: database system is ready to accept connections
I20260812 01:13:51.979391 21084 mini_postgres.cc:96] Postgres bound to 51097
2026-08-12 01:13:52.006 UTC [9622] FATAL: database "slave" does not exist
127.20.151.20:51097 - accepting connections
I20260812 01:13:52.007005 21084 mini_ranger.cc:161] Starting Ranger...
I20260812 01:13:52.043258 21084 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 01:13:52.540535 21084 mini_ranger.cc:87] Created ranger Postgres database
I20260812 01:13:52.540669 21084 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-admin
2026-08-12 01:13:52,850 [I] DB FLAVOR :POSTGRES
2026-08-12 01:13:52,851 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 01:13:52,851 [I] Checking connection..
2026-08-12 01:13:52,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 01:13:53,126 [I] Checking connection passed.
2026-08-12 01:13:53,126 [I] --------- Verifying version history table ---------
2026-08-12 01:13:53,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:13:53,398 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 01:13:53,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:13:53,646 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 01:13:53,646 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 01:13:53,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 01:13:53.899 UTC [9754] WARNING: there is no transaction in progress
2026-08-12 01:13:53,909 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 01:13:53,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:13:54,176 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 01:13:54,177 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 01:13:54,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 01:13:54,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 01:13:54,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-21b9.c.gcp-upstream.internal','N') ;"
2026-08-12 01:13:54,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 01:13:55,235 [I] Table x_portal_user does not exist in database ranger
2026-08-12 01:13:55,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 01:13:55,506 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 01:13:55,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 01:13:55,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 01:13:56,034 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 01:13:56,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 01:13:57.066 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.084 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.102 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.118 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.138 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.156 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.216 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.224 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.239 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.249 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.260 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.274 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.284 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.291 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.297 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57.305 UTC [9973] WARNING: there is no transaction in progress
2026-08-12 01:13:57,969 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 01:13:57,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 01:13:58,234 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 01:13:58,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 01:13:58,511 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 01:13:58,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 01:13:58,782 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 01:13:58,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 01:13:59,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 01:13:59,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-21b9.c.gcp-upstream.internal';"
2026-08-12 01:13:59,544 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 01:13:59,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 01:13:59,809 [I] DB_PATCHES have already been applied
I20260812 01:13:59.815517 21084 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWp05DU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 01:13:59.815614 21084 mini_ranger.cc:193] Using host: 127.20.151.20
I20260812 01:13:59.839731 21084 mini_ranger.cc:227] Ranger admin URL: http://127.20.151.20:38161
Aug 12, 2026 1:14:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 1:14:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 1:14:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 1:14:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 1:14:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 1:14:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 1:14:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 01:14:01.133940 21084 mini_ranger.cc:160] Time spent starting Ranger: real 9.127s user 0.005s sys 0.076s
Aug 12, 2026 1:14:22 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 1:14:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 1:14:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 1:14:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 1:14:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 01:14:23.745832 21084 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/rangerkms_127.20.151.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/rangerkms_127.20.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.151.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.151.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.151.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/HTTP_127.20.151.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.151.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/HTTP_127.20.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 01:14:23 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497263, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 01:14:23.997067 21084 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 01:14:24.034049 21084 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 01:14:25.039105 21084 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 01:14:25.039216 21084 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-kms
I20260812 01:14:25.039247 21084 mini_ranger_kms.cc:227] Using postgres at 127.20.151.20:51097
2026-08-12 01:14:25,437 [I] DB FLAVOR :POSTGRES
2026-08-12 01:14:25,437 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 01:14:25,437 [I] Checking connection
2026-08-12 01:14:25,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 01:14:25,696 [I] connection success
2026-08-12 01:14:25,696 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 01:14:25,696 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 01:14:25,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 01:14:25,946 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 01:14:25,946 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 01:14:25,947 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 01:14:25,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.151.20:51097/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 01:14:26,222 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 01:14:26.736581 21084 mini_ranger_kms.cc:310] Created kms service
I20260812 01:14:26.961438 21084 mini_ranger_kms.cc:326] Created kudu user
I20260812 01:14:27.012356 21084 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 01:14:27 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497263, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:14:27.436529 21084 mini_ranger_kms.cc:384] Added ranger policy
I20260812 01:14:27.436897 21084 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWp05DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWp05DU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 01:14:27.436973 21084 mini_ranger_kms.cc:240] Using host: 127.20.151.20
I20260812 01:14:27.462325 21084 mini_ranger_kms.cc:276] Ranger KMS PID: 10272
I20260812 01:14:27.462466 21084 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.151.20:44251
Aug 12, 2026 1:14:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 1:14:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 1:14:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 1:14:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 1:14:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 1:14:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 01:14:29 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786497269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.151.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:30.092324 21084 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.095s user 0.079s sys 0.371s
I20260812 01:14:30.092597 21084 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 01:14:30.092793 21084 mini_ranger_kms.cc:401] 127.20.151.20:44251/kms/v1/keys
Aug 12 01:14:30 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.151.20@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
Aug 12 01:14:30 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.151.20@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
Aug 12 01:14:32 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:14:33.029500 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46755
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:46755
--ranger_config_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:44251/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:14:33.395340 10344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:33.395660 10344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:33.395797 10344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:33.403153 10344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:14:33.403308 10344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:33.403390 10344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:33.403467 10344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:14:33.403532 10344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:14:33.414188 10344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.151.20:44251/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:46755
--ranger_config_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:33.415617 10344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:33.417327 10344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:33.429731 10361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:33.429762 10359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:33.429967 10357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:33.430601 10344 server_base.cc:1034] running on GCE node
I20260812 01:14:33.431490 10344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:33.432997 10344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:33.434247 10344 hybrid_clock.cc:648] HybridClock initialized: now 1786497273434193 us; error 36 us; skew 500 ppm
I20260812 01:14:33.434630 10344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:33 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497273, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:33.441599 10344 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:14:33.443238 10344 webserver.cc:460] Webserver started at http://127.20.151.62:35907/ using document root <none> and password file <none>
I20260812 01:14:33.443996 10344 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:33.444128 10344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:33.444463 10344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:14:33 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497273, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:14:33.499123 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "81c2deca005c4a38b91b338f67ca6aac"
format_stamp: "Formatted at 2026-08-12 01:14:33 on dist-test-slave-21b9"
server_key: "d5a0dcea3c02661928c7b0bfff7f02d7"
server_key_iv: "266baa5a6135816cc0fa5db0f4e61286"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:33.500104 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "81c2deca005c4a38b91b338f67ca6aac"
format_stamp: "Formatted at 2026-08-12 01:14:33 on dist-test-slave-21b9"
server_key: "d5a0dcea3c02661928c7b0bfff7f02d7"
server_key_iv: "266baa5a6135816cc0fa5db0f4e61286"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:33.506032 10344 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 01:14:33.523444 10370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:33.525058 10344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:14:33.525250 10344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "81c2deca005c4a38b91b338f67ca6aac"
format_stamp: "Formatted at 2026-08-12 01:14:33 on dist-test-slave-21b9"
server_key: "d5a0dcea3c02661928c7b0bfff7f02d7"
server_key_iv: "266baa5a6135816cc0fa5db0f4e61286"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:33.525498 10344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:33.566753 10344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:33.571127 10344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:33.571475 10344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:33.589882 10344 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:46755
I20260812 01:14:33.589901 10456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:46755 every 8 connection(s)
I20260812 01:14:33.591256 10344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:14:33.595075 10458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:33.596681 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 10344
I20260812 01:14:33.596925 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:14:33.597827 21084 external_mini_cluster.cc:1428] Setting key ff8af6c016284c3302ed9a95d55528fd
I20260812 01:14:33.610284 10458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac: Bootstrap starting.
Aug 12 01:14:33 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:33.614516 10458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:33.616014 10458 log.cc:824] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:33.620894 10458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac: No bootstrap required, opened a new log
I20260812 01:14:33.626209 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:33.600029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.20.151.62:46755)
0812 01:14:33.600883 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:33.600898 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:33.600953 (+ 55us) server_negotiation.cc:407] Connection header received
0812 01:14:33.602554 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:33.602577 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:33.602975 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:33.603339 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:33.605157 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:33.606481 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:33.608361 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:33.609578 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:33.609776 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:33.614929 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:33.614943 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:33.614965 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:33.615039 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:33.620697 (+ 5658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:33.621643 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:33.621650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:33.621657 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:33.621828 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:33.622261 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:33.622264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:33.622266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:33.622936 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:33.623192 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:33.624383 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0812 01:14:33.624840 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":418,"threads_started":1}
I20260812 01:14:33.630601 10458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } }
I20260812 01:14:33.630991 10458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:33.631069 10458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c2deca005c4a38b91b338f67ca6aac, State: Initialized, Role: FOLLOWER
I20260812 01:14:33.631652 10458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } }
I20260812 01:14:33.631865 10458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:14:33.631983 10458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:14:33.632112 10458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:33.634517 10458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } }
I20260812 01:14:33.635004 10458 leader_election.cc:302] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81c2deca005c4a38b91b338f67ca6aac; no voters:
I20260812 01:14:33.635656 10458 leader_election.cc:288] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:14:33.635785 10468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:14:33.636526 10468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 1 LEADER]: Becoming Leader. State: Replica: 81c2deca005c4a38b91b338f67ca6aac, State: Running, Role: LEADER
I20260812 01:14:33.637102 10468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } }
I20260812 01:14:33.637799 10458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:14:33.639870 10471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81c2deca005c4a38b91b338f67ca6aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } } }
I20260812 01:14:33.640308 10471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:33.639891 10473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81c2deca005c4a38b91b338f67ca6aac. Latest consensus state: current_term: 1 leader_uuid: "81c2deca005c4a38b91b338f67ca6aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } } }
I20260812 01:14:33.645174 10473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:33.648818 10478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:14:33.649135 10458 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 01:14:33.665107 10478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:14:33.676643 10478 catalog_manager.cc:1347] Generated new cluster ID: 54c321b7ce5543c288e99c0ee7dad620
I20260812 01:14:33.676790 10478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:14:33.696550 10478 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:14:33.697952 10478 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:14:33.714726 10478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac: Generated new TSK 0
I20260812 01:14:33.715907 10478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:14:34.527215 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:44251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46755
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:14:34 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786497274, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 01:14:34.995381 10531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:34.995918 10531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:34.996124 10531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:35.005566 10531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:35.005702 10531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:35.005857 10531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:14:35.016239 10531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.151.20:44251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:35.017584 10531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:35.019002 10531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:35.050578 10554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:35.052194 10557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:35.054101 10531 server_base.cc:1034] running on GCE node
W20260812 01:14:35.054575 10560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:35.055459 10531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:35.056782 10531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:35.058097 10531 hybrid_clock.cc:648] HybridClock initialized: now 1786497275058023 us; error 59 us; skew 500 ppm
I20260812 01:14:35.058506 10531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:35 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497275, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:35.062880 10531 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:14:35.072882 10531 webserver.cc:460] Webserver started at http://127.20.151.1:34943/ using document root <none> and password file <none>
I20260812 01:14:35.073866 10531 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:35.074021 10531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:35.074395 10531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:14:35 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497275, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:14:35.096771 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "f3697307d96847079aebacef625372a3"
format_stamp: "Formatted at 2026-08-12 01:14:35 on dist-test-slave-21b9"
server_key: "a9a2244f4372c309edfa9039736cf550"
server_key_iv: "41d8afbba0294f0c1e22e19082f7d4c0"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:35.098052 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "f3697307d96847079aebacef625372a3"
format_stamp: "Formatted at 2026-08-12 01:14:35 on dist-test-slave-21b9"
server_key: "a9a2244f4372c309edfa9039736cf550"
server_key_iv: "41d8afbba0294f0c1e22e19082f7d4c0"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:35.121516 10531 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.014s sys 0.003s
I20260812 01:14:35.156071 10579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:35.158392 10531 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.010s sys 0.003s
I20260812 01:14:35.158723 10531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "f3697307d96847079aebacef625372a3"
format_stamp: "Formatted at 2026-08-12 01:14:35 on dist-test-slave-21b9"
server_key: "a9a2244f4372c309edfa9039736cf550"
server_key_iv: "41d8afbba0294f0c1e22e19082f7d4c0"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:35.159125 10531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:35.189469 10531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:35.194810 10531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:35.195150 10531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:35.202772 10531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:35.211299 10531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:35.211510 10531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260812 01:14:35.211793 10531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:35.211941 10531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 01:14:35 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497274, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:14:35.318221 10531 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:41141
I20260812 01:14:35.318231 10786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:41141 every 8 connection(s)
I20260812 01:14:35.319893 10531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:14:35.321043 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 10531
I20260812 01:14:35.321282 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:14:35.322190 21084 external_mini_cluster.cc:1428] Setting key 83880e656958e923c7d0ba135946df7a
Aug 12 01:14:35 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497275, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:35.403055 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:35.333960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:48879 (local address 127.20.151.62:46755)
0812 01:14:35.363792 (+ 29832us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:35.363800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:35.365641 (+ 1841us) server_negotiation.cc:407] Connection header received
0812 01:14:35.371883 (+ 6242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:35.371890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:35.372010 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:35.372101 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:35.382711 (+ 10610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:35.384485 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:35.387979 (+ 3494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:35.388666 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:35.388777 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:35.393855 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:35.393863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:35.393868 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:35.393927 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:35.397999 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:35.399716 (+ 1717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:35.399723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:35.399726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:35.399908 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:35.400550 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:35.400556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:35.400559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:35.400916 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:35.401176 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:35.402380 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0812 01:14:35.402646 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29665,"thread_start_us":29525,"threads_started":1}
I20260812 01:14:35.405696 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:35.333314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46755 (local address 127.20.151.1:48879)
0812 01:14:35.365322 (+ 32008us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:35.365439 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:35.368929 (+ 3490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:35.372750 (+ 3821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:35.372763 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:35.373621 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:35.374780 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:35.374817 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:35.384891 (+ 10074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:35.384898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:35.387711 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:35.387733 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:35.389256 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:35.389267 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:35.389579 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:35.390459 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:35.390492 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:35.393537 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:35.398356 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:35.398369 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:35.398392 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:35.399535 (+ 1143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:35.400121 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:35.400125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:35.400129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:35.400410 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:35.401348 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:35.401357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:35.401858 (+ 501us) client_negotiation.cc:769] Sending connection context
0812 01:14:35.403597 (+ 1739us) client_negotiation.cc:240] Negotiation successful
0812 01:14:35.404237 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30836,"thread_start_us":30568,"threads_started":1}
I20260812 01:14:35.411705 10787 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46755
I20260812 01:14:35.412230 10787 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:35.413518 10787 heartbeater.cc:511] Master 127.20.151.62:46755 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
I20260812 01:14:35.417008 10397 ts_manager.cc:194] Registered new tserver with Master: f3697307d96847079aebacef625372a3 (127.20.151.1:41141)
I20260812 01:14:35.420082 10397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:48879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
W20260812 01:14:35.470525 21084 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.20.151.2: real 0.145s user 0.007s sys 0.025s
I20260812 01:14:35.471566 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:44251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46755
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:14:35 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497274, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
W20260812 01:14:35.942044 10798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:35.942369 10798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:35.942529 10798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:35.950059 10798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:35.950206 10798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:35.950366 10798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:14:35.960816 10798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.151.20:44251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:35.962244 10798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:35.963686 10798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:35.978609 10823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:35.979255 10821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:35.979509 10826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:35.979910 10798 server_base.cc:1034] running on GCE node
I20260812 01:14:35.980665 10798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:35.981900 10798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:35.983179 10798 hybrid_clock.cc:648] HybridClock initialized: now 1786497275983092 us; error 50 us; skew 500 ppm
I20260812 01:14:35.983589 10798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:35 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497275, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:35.987619 10798 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:14:35.989527 10798 webserver.cc:460] Webserver started at http://127.20.151.2:34065/ using document root <none> and password file <none>
I20260812 01:14:35.990443 10798 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:35.990588 10798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:35.990954 10798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:14:35 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497275, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:14:36.010161 10798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "27b688bc7851487a9e7d8f8251a71c4a"
format_stamp: "Formatted at 2026-08-12 01:14:36 on dist-test-slave-21b9"
server_key: "d97e6b7b357b3312805552fd6f926a75"
server_key_iv: "5e496d6f3a90db82844c6ea737c3d17b"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:36.011561 10798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "27b688bc7851487a9e7d8f8251a71c4a"
format_stamp: "Formatted at 2026-08-12 01:14:36 on dist-test-slave-21b9"
server_key: "d97e6b7b357b3312805552fd6f926a75"
server_key_iv: "5e496d6f3a90db82844c6ea737c3d17b"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:36.022541 10798 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.008s sys 0.000s
I20260812 01:14:36.053157 10840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:36.055155 10798 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.000s
I20260812 01:14:36.055397 10798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "27b688bc7851487a9e7d8f8251a71c4a"
format_stamp: "Formatted at 2026-08-12 01:14:36 on dist-test-slave-21b9"
server_key: "d97e6b7b357b3312805552fd6f926a75"
server_key_iv: "5e496d6f3a90db82844c6ea737c3d17b"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:36.055728 10798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:36.084659 10798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:36.088930 10798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:36.089273 10798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:36.090340 10798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:36.091918 10798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:36.092001 10798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:36.092125 10798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:36.092183 10798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:36.129578 10798 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:40963
I20260812 01:14:36.129595 11032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:40963 every 8 connection(s)
I20260812 01:14:36.130980 10798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:14:36.135475 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 10798
I20260812 01:14:36.135654 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:14:36.136229 21084 external_mini_cluster.cc:1428] Setting key f35441511f511938aa7f78d745b8405f
Aug 12 01:14:36 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497275, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:36.155402 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:36.133819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:48363 (local address 127.20.151.62:46755)
0812 01:14:36.134296 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:36.134301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:36.136865 (+ 2564us) server_negotiation.cc:407] Connection header received
0812 01:14:36.137841 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:36.137847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:36.137957 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:36.138100 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:36.140059 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.141102 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.143301 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.143919 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.144053 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:36.148524 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:36.148530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:36.148534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:36.148575 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:36.151802 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:36.152776 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:36.152782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:36.152785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:36.152931 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:36.153482 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:36.153487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:36.153490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:36.153819 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:36.153966 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:36.154736 (+ 770us) server_negotiation.cc:299] Negotiation successful
0812 01:14:36.154980 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":256,"threads_started":1}
I20260812 01:14:36.156502 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:36.134484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46755 (local address 127.20.151.2:48363)
0812 01:14:36.135344 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:36.135393 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:36.137566 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:36.138360 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:36.138369 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:36.138933 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:36.139849 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:36.139876 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.141293 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.141298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:36.143104 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:36.143122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.144182 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.144189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:36.144498 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:36.145331 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:36.145362 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:36.148227 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:36.151974 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:36.151985 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:36.152005 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:36.152628 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:36.153049 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:36.153054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:36.153058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:36.153347 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:36.153968 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:36.153986 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:36.154339 (+ 353us) client_negotiation.cc:769] Sending connection context
0812 01:14:36.154598 (+ 259us) client_negotiation.cc:240] Negotiation successful
0812 01:14:36.155176 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":318,"threads_started":1}
I20260812 01:14:36.158061 11034 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46755
I20260812 01:14:36.158370 11034 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:36.159117 11034 heartbeater.cc:511] Master 127.20.151.62:46755 requested a full tablet report, sending...
I20260812 01:14:36.160681 10397 ts_manager.cc:194] Registered new tserver with Master: 27b688bc7851487a9e7d8f8251a71c4a (127.20.151.2:40963)
I20260812 01:14:36.161610 10397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:48363
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:14:36.223061 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:44251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46755
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:14:36.425841 10787 heartbeater.cc:503] Master 127.20.151.62:46755 was elected leader, sending a full tablet report...
W20260812 01:14:36.606850 11045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:36.607160 11045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:36.607333 11045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:36.614820 11045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:36.614970 11045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:36.615139 11045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:14:36.625329 11045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.151.20:44251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:36.626736 11045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:36.628096 11045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:36.641595 11051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:36.641485 11058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:36.641537 11052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:36.642590 11045 server_base.cc:1034] running on GCE node
I20260812 01:14:36.643232 11045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:36.644286 11045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:36.645551 11045 hybrid_clock.cc:648] HybridClock initialized: now 1786497276645506 us; error 35 us; skew 500 ppm
I20260812 01:14:36.645896 11045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:36 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:36.649194 11045 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:14:36.650707 11045 webserver.cc:460] Webserver started at http://127.20.151.3:34417/ using document root <none> and password file <none>
I20260812 01:14:36.651508 11045 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:36.651636 11045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:36.651965 11045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:14:36 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:14:36.666923 11045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "f78239b47b824458ad4fbc2efd1c7fb8"
format_stamp: "Formatted at 2026-08-12 01:14:36 on dist-test-slave-21b9"
server_key: "a88f2834394fe6595ff5cd25380eb691"
server_key_iv: "a3aaebf8f17ef5369c676d736cfd3e37"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:36.667912 11045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "f78239b47b824458ad4fbc2efd1c7fb8"
format_stamp: "Formatted at 2026-08-12 01:14:36 on dist-test-slave-21b9"
server_key: "a88f2834394fe6595ff5cd25380eb691"
server_key_iv: "a3aaebf8f17ef5369c676d736cfd3e37"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:36.673771 11045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 01:14:36.688977 11068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:36.690474 11045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 01:14:36.690654 11045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "f78239b47b824458ad4fbc2efd1c7fb8"
format_stamp: "Formatted at 2026-08-12 01:14:36 on dist-test-slave-21b9"
server_key: "a88f2834394fe6595ff5cd25380eb691"
server_key_iv: "a3aaebf8f17ef5369c676d736cfd3e37"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:36.690897 11045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:36.706557 11045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:36.710758 11045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:36.711055 11045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:36.712216 11045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:36.713699 11045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:36.713778 11045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:36.713898 11045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:36.713949 11045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:36.748765 11045 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:44103
I20260812 01:14:36.748807 11265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:44103 every 8 connection(s)
I20260812 01:14:36.750087 11045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:14:36.754490 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 11045
I20260812 01:14:36.754701 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:14:36.755192 21084 external_mini_cluster.cc:1428] Setting key 82a5021e1365cc7375dfe70f12249cbb
Aug 12 01:14:36 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497276, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:36.770896 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:36.752837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:37361 (local address 127.20.151.62:46755)
0812 01:14:36.753403 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:36.753408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:36.754324 (+ 916us) server_negotiation.cc:407] Connection header received
0812 01:14:36.756421 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:36.756427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:36.756519 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:36.756643 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:36.758322 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.759114 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.760687 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.761139 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.761243 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:36.764612 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:36.764619 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:36.764623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:36.764691 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:36.767410 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:36.768213 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:36.768217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:36.768220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:36.768379 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:36.768836 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:36.768840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:36.768842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:36.769208 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:36.769357 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:36.770296 (+ 939us) server_negotiation.cc:299] Negotiation successful
0812 01:14:36.770515 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":240,"threads_started":1}
I20260812 01:14:36.771924 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:36.753425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46755 (local address 127.20.151.3:37361)
0812 01:14:36.754155 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:36.754194 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:36.756151 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:36.756933 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:36.756945 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:36.757518 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:36.758144 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:36.758171 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.759288 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.759294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:36.760563 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:36.760576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.761331 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.761335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:36.761562 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:36.762254 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:36.762287 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:36.764390 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:36.767547 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:36.767555 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:36.767570 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:36.768076 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:36.768471 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:36.768474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:36.768477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:36.768716 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:36.769354 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:36.769362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:36.769770 (+ 408us) client_negotiation.cc:769] Sending connection context
0812 01:14:36.770119 (+ 349us) client_negotiation.cc:240] Negotiation successful
0812 01:14:36.770653 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":298,"threads_started":1}
I20260812 01:14:36.773372 11266 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46755
I20260812 01:14:36.773699 11266 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:36.774406 11266 heartbeater.cc:511] Master 127.20.151.62:46755 requested a full tablet report, sending...
I20260812 01:14:36.775910 10397 ts_manager.cc:194] Registered new tserver with Master: f78239b47b824458ad4fbc2efd1c7fb8 (127.20.151.3:44103)
I20260812 01:14:36.776839 10397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:37361
I20260812 01:14:36.783447 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:14:36.785182 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 10531
Aug 12 01:14:36 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:14:36.801858 11278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:36.786120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.20.151.1:41141)
0812 01:14:36.786724 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:36.786731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:36.786757 (+ 26us) server_negotiation.cc:407] Connection header received
0812 01:14:36.786821 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:36.786828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:36.787035 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:36.787177 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:36.788608 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.789678 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.791140 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.791771 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.791869 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:36.796266 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:36.796282 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:36.796297 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:36.796336 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:36.799264 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:36.800042 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:36.800049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:36.800052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:36.800225 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:36.800515 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:36.800517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:36.800519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:36.800873 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:36.801114 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:36.801350 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 01:14:36.801563 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":294,"threads_started":1}
I20260812 01:14:36.803172 10772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:36.888218 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 10531
I20260812 01:14:36.913403 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 10798
Aug 12 01:14:36 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:14:36.935222 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:36.914387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.20.151.2:40963)
0812 01:14:36.914931 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:36.914939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:36.914965 (+ 26us) server_negotiation.cc:407] Connection header received
0812 01:14:36.915050 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:36.915057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:36.915404 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:36.915562 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:36.916852 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.917875 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.919361 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:36.919968 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:36.920088 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:36.929248 (+ 9160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:36.929265 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:36.929284 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:36.929342 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:36.932251 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:36.932997 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:36.933005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:36.933007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:36.933155 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:36.933670 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:36.933673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:36.933675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:36.934039 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:36.934268 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:36.934697 (+ 429us) server_negotiation.cc:299] Negotiation successful
0812 01:14:36.934917 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":284,"threads_started":1}
I20260812 01:14:36.936538 11027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:37.020382 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 10798
I20260812 01:14:37.042698 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 11045
Aug 12 01:14:37 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:14:37.059762 11286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:37.043735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.20.151.3:44103)
0812 01:14:37.044393 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:37.044402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:37.044437 (+ 35us) server_negotiation.cc:407] Connection header received
0812 01:14:37.044502 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:37.044509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:37.044736 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:37.044886 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:37.046219 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:37.047256 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:37.048617 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:37.049223 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:37.049342 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:37.053835 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:37.053850 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:37.053866 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:37.053910 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:37.056705 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:37.057479 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:37.057486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:37.057489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:37.057637 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:37.058043 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:37.058047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:37.058050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:37.058419 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:37.058731 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:37.059080 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 01:14:37.059382 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":338,"threads_started":1}
I20260812 01:14:37.061090 11254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:37.145490 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 11045
I20260812 01:14:37.172071 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 10344
I20260812 01:14:37.173527 10450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:37.286065 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 10344
I20260812 01:14:37.322088 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46755
--webserver_interface=127.20.151.62
--webserver_port=35907
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:46755
--ranger_config_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:44251/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.151.20:44251/kms with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:14:37.668991 11293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:37.669353 11293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:37.669499 11293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:37.677350 11293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:14:37.677490 11293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:37.677579 11293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:37.677657 11293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:14:37.677728 11293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:14:37.688331 11293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.151.20:44251/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:46755
--ranger_config_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=35907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:37.689725 11293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:37.691087 11293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:37.702941 11299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:37.703125 11302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:37.703879 11293 server_base.cc:1034] running on GCE node
W20260812 01:14:37.704027 11300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:37.704927 11293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:37.706362 11293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:37.707567 11293 hybrid_clock.cc:648] HybridClock initialized: now 1786497277707502 us; error 53 us; skew 500 ppm
I20260812 01:14:37.707942 11293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:37 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497277, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:37.711220 11293 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:14:37.712837 11293 webserver.cc:460] Webserver started at http://127.20.151.62:35907/ using document root <none> and password file <none>
I20260812 01:14:37.713518 11293 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:37.713649 11293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 01:14:37 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497277, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:14:37.796761 11293 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 01:14:37.800290 11314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:37.801715 11293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 01:14:37.801898 11293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "81c2deca005c4a38b91b338f67ca6aac"
format_stamp: "Formatted at 2026-08-12 01:14:33 on dist-test-slave-21b9"
server_key: "d5a0dcea3c02661928c7b0bfff7f02d7"
server_key_iv: "266baa5a6135816cc0fa5db0f4e61286"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:37.802902 11293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:37.817826 11293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:37.822026 11293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:37.822316 11293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:37.841061 11293 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:46755
I20260812 01:14:37.841082 11398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:46755 every 8 connection(s)
I20260812 01:14:37.842617 11293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:14:37.845486 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 11293
I20260812 01:14:37.846354 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:44251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:41141
--local_ip_for_outbound_sockets=127.20.151.1
--tserver_master_addrs=127.20.151.62:46755
--webserver_port=34943
--webserver_interface=127.20.151.1
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.151.20:44251/kms with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:14:37.849979 11400 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:14:37.853482 11400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac: Bootstrap starting.
I20260812 01:14:37.873160 11400 log.cc:824] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:37.882866 11400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:14:37.883497 11400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac: Bootstrap complete.
I20260812 01:14:37.892797 11400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } }
I20260812 01:14:37.893719 11400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c2deca005c4a38b91b338f67ca6aac, State: Initialized, Role: FOLLOWER
I20260812 01:14:37.894361 11400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } }
I20260812 01:14:37.894595 11400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:14:37.894712 11400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:14:37.894882 11400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 1 FOLLOWER]: Advancing to term 2
I20260812 01:14:37.899513 11400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } }
I20260812 01:14:37.900033 11400 leader_election.cc:302] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81c2deca005c4a38b91b338f67ca6aac; no voters:
I20260812 01:14:37.900656 11400 leader_election.cc:288] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 01:14:37.900916 11404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 2 FOLLOWER]: Leader election won for term 2
I20260812 01:14:37.901757 11404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [term 2 LEADER]: Becoming Leader. State: Replica: 81c2deca005c4a38b91b338f67ca6aac, State: Running, Role: LEADER
I20260812 01:14:37.902415 11404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } }
I20260812 01:14:37.902623 11400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:14:37.905884 11408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81c2deca005c4a38b91b338f67ca6aac. Latest consensus state: current_term: 2 leader_uuid: "81c2deca005c4a38b91b338f67ca6aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } } }
I20260812 01:14:37.906302 11408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:37.906584 11406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "81c2deca005c4a38b91b338f67ca6aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c2deca005c4a38b91b338f67ca6aac" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46755 } } }
I20260812 01:14:37.906874 11406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:37.918110 11419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:14:37.924541 11419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:14:37.926052 11419 catalog_manager.cc:1259] Loaded cluster ID: 54c321b7ce5543c288e99c0ee7dad620
I20260812 01:14:37.926252 11419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:14:37.930308 11419 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:14:37.932224 11419 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac: Loaded TSK: 0
I20260812 01:14:37.933893 11419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:14:37.937161 11400 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260812 01:14:38.279850 11402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:38.280215 11402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:38.280397 11402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:38.291060 11402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:38.291244 11402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:38.291498 11402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:14:38.307403 11402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.151.20:44251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:41141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=34943
--tserver_master_addrs=127.20.151.62:46755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:38.309460 11402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:38.311396 11402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:38.334226 11465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:38.343735 11466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:38.350870 11473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:38.352108 11402 server_base.cc:1034] running on GCE node
I20260812 01:14:38.352989 11402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:38.354275 11402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:38.355592 11402 hybrid_clock.cc:648] HybridClock initialized: now 1786497278355512 us; error 65 us; skew 500 ppm
I20260812 01:14:38.355983 11402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:38 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497278, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:38.359915 11402 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:14:38.383801 11402 webserver.cc:460] Webserver started at http://127.20.151.1:34943/ using document root <none> and password file <none>
I20260812 01:14:38.384665 11402 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:38.384837 11402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 01:14:38 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497278, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:14:38.419061 11402 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.005s sys 0.005s
I20260812 01:14:38.431176 11483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:38.433599 11402 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.004s sys 0.002s
I20260812 01:14:38.433954 11402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "f3697307d96847079aebacef625372a3"
format_stamp: "Formatted at 2026-08-12 01:14:35 on dist-test-slave-21b9"
server_key: "a9a2244f4372c309edfa9039736cf550"
server_key_iv: "41d8afbba0294f0c1e22e19082f7d4c0"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:38.436234 11402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:38.481547 11402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:38.487217 11402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:38.487735 11402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:38.489295 11402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:38.496224 11402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:38.496490 11402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 01:14:38.496713 11402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:38.496862 11402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:38.570123 11402 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:41141
I20260812 01:14:38.570236 11673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:41141 every 8 connection(s)
I20260812 01:14:38.571889 11402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:14:38.573184 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 11402
I20260812 01:14:38.574173 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:44251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:40963
--local_ip_for_outbound_sockets=127.20.151.2
--tserver_master_addrs=127.20.151.62:46755
--webserver_port=34065
--webserver_interface=127.20.151.2
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.151.20:44251/kms with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:14:38 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497278, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:38.619688 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:38.579608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:53695 (local address 127.20.151.62:46755)
0812 01:14:38.591701 (+ 12093us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:38.591727 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:38.594273 (+ 2546us) server_negotiation.cc:407] Connection header received
0812 01:14:38.594819 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:38.594856 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:38.595440 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:38.595872 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:38.598494 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:38.600511 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:38.602219 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:38.603136 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:38.603377 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:38.608564 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:38.608578 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:38.608601 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:38.608657 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:38.612886 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:38.614149 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:38.614161 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:38.614170 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:38.614446 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:38.615115 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:38.615122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:38.615126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:38.615938 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:38.616188 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:38.617345 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0812 01:14:38.617846 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11748,"thread_start_us":11507,"threads_started":1}
I20260812 01:14:38.625715 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:38.576825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46755 (local address 127.20.151.1:53695)
0812 01:14:38.590567 (+ 13742us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:38.591840 (+ 1273us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:38.593876 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:38.596648 (+ 2772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:38.596661 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:38.597375 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:38.598210 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:38.598242 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:38.600737 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:38.600741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:38.602037 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:38.602053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:38.603918 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:38.603927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:38.604145 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:38.604992 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:38.605032 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:38.608194 (+ 3162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:38.613127 (+ 4933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:38.613140 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:38.613159 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:38.613911 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:38.614630 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:38.614636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:38.614640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:38.614953 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:38.616161 (+ 1208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:38.616170 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:38.616673 (+ 503us) client_negotiation.cc:769] Sending connection context
0812 01:14:38.617032 (+ 359us) client_negotiation.cc:240] Negotiation successful
0812 01:14:38.624276 (+ 7244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13298,"thread_start_us":12994,"threads_started":1}
I20260812 01:14:38.629864 11674 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46755
I20260812 01:14:38.630230 11674 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:38.631186 11674 heartbeater.cc:511] Master 127.20.151.62:46755 requested a full tablet report, sending...
W20260812 01:14:38.634591 11674 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:46755 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 01:14:38.634825 11674 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:38.635138 11674 heartbeater.cc:511] Master 127.20.151.62:46755 requested a full tablet report, sending...
I20260812 01:14:38.636395 11674 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:38.636667 11674 heartbeater.cc:511] Master 127.20.151.62:46755 requested a full tablet report, sending...
W20260812 01:14:38.637741 11674 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 01:14:39.024101 11687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:39.024394 11687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:39.024516 11687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:39.034072 11687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:39.034292 11687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:39.034494 11687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:14:39.046876 11687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.151.20:44251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:40963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=34065
--tserver_master_addrs=127.20.151.62:46755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:39.048703 11687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:39.050495 11687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:39.067657 11710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:39.067705 11711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:39.073556 11687 server_base.cc:1034] running on GCE node
W20260812 01:14:39.075574 11715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:39.076408 11687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:39.077639 11687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:39.078881 11687 hybrid_clock.cc:648] HybridClock initialized: now 1786497279078827 us; error 44 us; skew 500 ppm
I20260812 01:14:39.079296 11687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:39 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:39.082707 11687 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:14:39.084594 11687 webserver.cc:460] Webserver started at http://127.20.151.2:34065/ using document root <none> and password file <none>
I20260812 01:14:39.085352 11687 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:39.085495 11687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 01:14:39 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:14:39.110473 11687 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260812 01:14:39.122256 11729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:39.128218 11687 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.007s sys 0.004s
I20260812 01:14:39.128563 11687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "27b688bc7851487a9e7d8f8251a71c4a"
format_stamp: "Formatted at 2026-08-12 01:14:36 on dist-test-slave-21b9"
server_key: "d97e6b7b357b3312805552fd6f926a75"
server_key_iv: "5e496d6f3a90db82844c6ea737c3d17b"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:39.130287 11687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:39.155263 11687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:39.160995 11687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:39.161410 11687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:39.167807 11687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:39.170166 11687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:39.170305 11687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:39.170476 11687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:39.170557 11687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 01:14:39 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786497279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:39.226545 11902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:40963 every 8 connection(s)
I20260812 01:14:39.226893 11687 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:40963
I20260812 01:14:39.228935 11687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:14:39.229887 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 11687
I20260812 01:14:39.230664 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.151.20:44251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:44103
--local_ip_for_outbound_sockets=127.20.151.3
--tserver_master_addrs=127.20.151.62:46755
--webserver_port=34417
--webserver_interface=127.20.151.3
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.151.20:44251/kms with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:14:39 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:39.287986 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:39.246461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:47211 (local address 127.20.151.62:46755)
0812 01:14:39.263732 (+ 17271us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:39.263740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:39.266802 (+ 3062us) server_negotiation.cc:407] Connection header received
0812 01:14:39.266873 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:39.266877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:39.266948 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:39.267050 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:39.269207 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.270261 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.272984 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.273591 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.273707 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:39.280302 (+ 6595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:39.280310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:39.280315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:39.280365 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:39.284144 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:39.285231 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:39.285237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:39.285241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:39.285413 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:39.285938 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:39.285943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:39.285947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:39.286313 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:39.286460 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:39.287359 (+ 899us) server_negotiation.cc:299] Negotiation successful
0812 01:14:39.287612 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17101,"thread_start_us":16982,"threads_started":1}
I20260812 01:14:39.289155 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:39.245775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46755 (local address 127.20.151.2:47211)
0812 01:14:39.264362 (+ 18587us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:39.264424 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:39.266470 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:39.267419 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:39.267431 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:39.268061 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:39.268899 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:39.268934 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.270453 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.270458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:39.272526 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:39.272570 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.274043 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.274051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:39.274326 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:39.276989 (+ 2663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:39.277026 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:39.279935 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:39.284361 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:39.284373 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:39.284396 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:39.285062 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:39.285531 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:39.285536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:39.285540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:39.285819 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:39.286440 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:39.286449 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:39.286899 (+ 450us) client_negotiation.cc:769] Sending connection context
0812 01:14:39.287203 (+ 304us) client_negotiation.cc:240] Negotiation successful
0812 01:14:39.287800 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18270,"thread_start_us":18018,"threads_started":1}
I20260812 01:14:39.290868 11903 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46755
I20260812 01:14:39.291214 11903 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:39.292214 11903 heartbeater.cc:511] Master 127.20.151.62:46755 requested a full tablet report, sending...
I20260812 01:14:39.294426 11344 ts_manager.cc:194] Registered new tserver with Master: 27b688bc7851487a9e7d8f8251a71c4a (127.20.151.2:40963)
I20260812 01:14:39.296680 11344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:47211
I20260812 01:14:39.637883 11674 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:39.638367 11674 heartbeater.cc:511] Master 127.20.151.62:46755 requested a full tablet report, sending...
I20260812 01:14:39.639781 11344 ts_manager.cc:194] Registered new tserver with Master: f3697307d96847079aebacef625372a3 (127.20.151.1:41141)
I20260812 01:14:39.640880 11344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:53695
Aug 12 01:14:39 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
W20260812 01:14:39.720361 11906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:39.720928 11906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:39.721149 11906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:39.732338 11906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:39.732602 11906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:39.732851 11906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:14:39.748637 11906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.151.20:44251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:44103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=34417
--tserver_master_addrs=127.20.151.62:46755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:39.750501 11906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:39.752341 11906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:39.770339 11924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:39.770329 11926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:39.775484 11928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:39.776897 11906 server_base.cc:1034] running on GCE node
I20260812 01:14:39.777721 11906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:39.779034 11906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:39.780333 11906 hybrid_clock.cc:648] HybridClock initialized: now 1786497279780247 us; error 75 us; skew 500 ppm
I20260812 01:14:39.780730 11906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:39 dist-test-slave-21b9 krb5kdc[9557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:39.785115 11906 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:14:39.787302 11906 webserver.cc:460] Webserver started at http://127.20.151.3:34417/ using document root <none> and password file <none>
I20260812 01:14:39.788251 11906 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:39.788407 11906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 01:14:39 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:14:39.814837 11906 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 01:14:39.818545 11940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:39.820214 11906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 01:14:39.820392 11906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "f78239b47b824458ad4fbc2efd1c7fb8"
format_stamp: "Formatted at 2026-08-12 01:14:36 on dist-test-slave-21b9"
server_key: "a88f2834394fe6595ff5cd25380eb691"
server_key_iv: "a3aaebf8f17ef5369c676d736cfd3e37"
server_key_version: "kuduclusterkey@0"
I20260812 01:14:39.821451 11906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:39.844534 11906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:39.849852 11906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:39.850217 11906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:39.851435 11906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:39.852978 11906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:39.853088 11906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:39.853211 11906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:39.853291 11906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:39.908219 11906 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:44103
I20260812 01:14:39.908267 12135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:44103 every 8 connection(s)
I20260812 01:14:39.909974 11906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:14:39.920001 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 11906
Aug 12 01:14:39 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:39.956509 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:39.913829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46755 (local address 127.20.151.3:40021)
0812 01:14:39.919998 (+ 6169us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:39.920079 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:39.922783 (+ 2704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:39.935741 (+ 12958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:39.935756 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:39.936477 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:39.937454 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:39.937490 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.939217 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.939223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:39.941334 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:39.941352 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.943832 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.943839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:39.944138 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:39.944944 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:39.944974 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:39.947311 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:39.951380 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:39.951391 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:39.951412 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:39.952050 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:39.952526 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:39.952530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:39.952534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:39.952795 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:39.953546 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:39.953555 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:39.954031 (+ 476us) client_negotiation.cc:769] Sending connection context
0812 01:14:39.954360 (+ 329us) client_negotiation.cc:240] Negotiation successful
0812 01:14:39.954979 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5773,"thread_start_us":6970,"threads_started":1}
I20260812 01:14:39.957731 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:39.914570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:40021 (local address 127.20.151.62:46755)
0812 01:14:39.923608 (+ 9038us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:39.923614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:39.923637 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:14:39.923704 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:39.923709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:39.923814 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:39.923925 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:39.937775 (+ 13850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.938974 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.942616 (+ 3642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.943395 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.943514 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:39.947608 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:39.947614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:39.947619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:39.947661 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:39.951192 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:39.952229 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:39.952234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:39.952238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:39.952386 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:39.952958 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:39.952963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:39.952966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:39.953322 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:39.953455 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:39.957107 (+ 3652us) server_negotiation.cc:299] Negotiation successful
0812 01:14:39.957345 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8894,"thread_start_us":8753,"threads_started":1}
I20260812 01:14:39.959163 12136 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46755
I20260812 01:14:39.959515 12136 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:39.960383 12136 heartbeater.cc:511] Master 127.20.151.62:46755 requested a full tablet report, sending...
I20260812 01:14:39.962226 11344 ts_manager.cc:194] Registered new tserver with Master: f78239b47b824458ad4fbc2efd1c7fb8 (127.20.151.3:44103)
I20260812 01:14:39.963366 11344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:40021
I20260812 01:14:39.963740 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:39.923563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.20.151.62:46755)
0812 01:14:39.942660 (+ 19097us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:39.942667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:39.942691 (+ 24us) server_negotiation.cc:407] Connection header received
0812 01:14:39.942752 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:39.942757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:39.942886 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:39.943003 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:39.945412 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.946577 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.949174 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.949865 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.949996 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:39.953670 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:39.953677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:39.953681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:39.953716 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:39.957502 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:39.958972 (+ 1470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:39.958978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:39.958981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:39.961055 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:39.961835 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:39.961841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:39.961844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:39.962254 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:39.962551 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:39.963016 (+ 465us) server_negotiation.cc:299] Negotiation successful
0812 01:14:39.963251 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2117,"server-negotiator.queue_time_us":18953,"thread_start_us":18815,"threads_started":1}
I20260812 01:14:39.966272 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:14:39.989080 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:39.970253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.20.151.62:46755)
0812 01:14:39.971334 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:39.971341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:39.972009 (+ 668us) server_negotiation.cc:407] Connection header received
0812 01:14:39.972292 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:39.972298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:39.972416 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:39.972535 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:39.974721 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.975834 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.977646 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:39.978274 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:39.978378 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:39.982275 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:39.982283 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:39.982288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:39.982336 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:39.985725 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:39.986627 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:39.986632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:39.986636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:39.986798 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:39.987238 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:39.987243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:39.987246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:39.987623 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:39.987854 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:39.988464 (+ 610us) server_negotiation.cc:299] Negotiation successful
0812 01:14:39.988733 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898}
I20260812 01:14:39.995667 11344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:14:40.009690 11344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 12 01:14:40 dist-test-slave-21b9 krb5kdc[9557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497279, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for HTTP/127.20.151.20@KRBTEST.COM
I20260812 01:14:40.107249 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.077952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:44103 (local address 127.0.0.1:48536)
0812 01:14:40.084720 (+ 6768us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.084738 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:40.084848 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:40.098247 (+ 13399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:40.098256 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:40.098307 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:40.098803 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:40.098823 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.100960 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.100965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:40.103654 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:40.103671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.105738 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.105746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:40.105970 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.106044 (+ 74us) client_negotiation.cc:769] Sending connection context
0812 01:14:40.106178 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 01:14:40.106814 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6651,"mutex_wait_us":24,"thread_start_us":7361,"threads_started":1}
I20260812 01:14:40.108790 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.078709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.20.151.3:44103)
0812 01:14:40.097425 (+ 18716us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:40.097446 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:40.097486 (+ 40us) server_negotiation.cc:407] Connection header received
0812 01:14:40.097608 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:40.097617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:40.097921 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:40.098167 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:40.099069 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.100774 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.103871 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.105468 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.107832 (+ 2364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.108120 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:40.108205 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 01:14:40.108396 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18512,"thread_start_us":18337,"threads_started":1}
I20260812 01:14:40.112672 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.076222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:40963 (local address 127.0.0.1:49742)
0812 01:14:40.083930 (+ 7708us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.083973 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:40.084175 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:40.098735 (+ 14560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:40.098741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:40.098774 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:40.099143 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:40.099156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.102017 (+ 2861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.102024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:40.104159 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:40.104176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.111752 (+ 7576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.111761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:40.111986 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.112027 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 01:14:40.112115 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 01:14:40.112269 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7407,"mutex_wait_us":3,"thread_start_us":7244,"threads_started":1}
I20260812 01:14:40.113576 12021 tablet_service.cc:1467] Processing CreateTablet for tablet 4811318039b0411eaa98ca65673145a3 (DEFAULT_TABLE table=test-table [id=3cdbd84dd03d4f20af678e6e670897cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:14:40.114307 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.076276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.20.151.2:40963)
0812 01:14:40.098136 (+ 21860us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:40.098144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:40.098170 (+ 26us) server_negotiation.cc:407] Connection header received
0812 01:14:40.098272 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:40.098278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:40.098510 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:40.098733 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:40.099384 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.100993 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.104410 (+ 3417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.105863 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.113449 (+ 7586us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.113706 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:40.113785 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 01:14:40.113982 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21706,"thread_start_us":22638,"threads_started":1}
I20260812 01:14:40.115873 12021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4811318039b0411eaa98ca65673145a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:40.119897 11806 tablet_service.cc:1467] Processing CreateTablet for tablet 4811318039b0411eaa98ca65673145a3 (DEFAULT_TABLE table=test-table [id=3cdbd84dd03d4f20af678e6e670897cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:14:40.121546 11806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4811318039b0411eaa98ca65673145a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:40.139163 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.085132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41141 (local address 127.0.0.1:37438)
0812 01:14:40.094544 (+ 9412us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.094569 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:40.094732 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:40.131585 (+ 36853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:40.131591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:40.131636 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:40.131983 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:40.131998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.134296 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.134302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:40.136353 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:40.136375 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.138364 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.138370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:40.138569 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.138608 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 01:14:40.138676 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 01:14:40.138814 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9264,"thread_start_us":9689,"threads_started":1}
I20260812 01:14:40.140674 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.085452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.20.151.1:41141)
0812 01:14:40.115021 (+ 29569us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:40.115030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:40.115059 (+ 29us) server_negotiation.cc:407] Connection header received
0812 01:14:40.115165 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:40.115173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:40.115460 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:40.132225 (+ 16765us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:40.132519 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.134112 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.136709 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.138125 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.139829 (+ 1704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.140094 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:40.140171 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 01:14:40.140348 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29384,"thread_start_us":30161,"threads_started":1}
I20260812 01:14:40.145649 11563 tablet_service.cc:1467] Processing CreateTablet for tablet 4811318039b0411eaa98ca65673145a3 (DEFAULT_TABLE table=test-table [id=3cdbd84dd03d4f20af678e6e670897cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:14:40.147653 11563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4811318039b0411eaa98ca65673145a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:40.155206 12182 tablet_bootstrap.cc:492] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: Bootstrap starting.
I20260812 01:14:40.159821 12181 tablet_bootstrap.cc:492] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8: Bootstrap starting.
I20260812 01:14:40.160396 12182 tablet_bootstrap.cc:654] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:40.163681 12182 log.cc:824] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:40.163960 12181 tablet_bootstrap.cc:654] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:40.165311 12181 log.cc:824] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:40.173934 12182 tablet_bootstrap.cc:492] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: No bootstrap required, opened a new log
I20260812 01:14:40.174621 12182 ts_tablet_manager.cc:1397] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20260812 01:14:40.188105 12181 tablet_bootstrap.cc:492] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8: No bootstrap required, opened a new log
I20260812 01:14:40.188442 12185 tablet_bootstrap.cc:492] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3: Bootstrap starting.
I20260812 01:14:40.188500 12181 ts_tablet_manager.cc:1397] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.009s
I20260812 01:14:40.187332 12182 raft_consensus.cc:348] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b688bc7851487a9e7d8f8251a71c4a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40963 } } peers { permanent_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44103 } } peers { permanent_uuid: "f3697307d96847079aebacef625372a3" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41141 } }
I20260812 01:14:40.191855 12182 raft_consensus.cc:374] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:40.192025 12182 raft_consensus.cc:729] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b688bc7851487a9e7d8f8251a71c4a, State: Initialized, Role: FOLLOWER
I20260812 01:14:40.192874 12182 consensus_queue.cc:260] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b688bc7851487a9e7d8f8251a71c4a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40963 } } peers { permanent_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44103 } } peers { permanent_uuid: "f3697307d96847079aebacef625372a3" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41141 } }
I20260812 01:14:40.194788 12185 tablet_bootstrap.cc:654] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:40.196287 12185 log.cc:824] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:40.205214 12182 ts_tablet_manager.cc:1428] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: Time spent starting tablet: real 0.030s user 0.016s sys 0.000s
I20260812 01:14:40.205884 11903 heartbeater.cc:503] Master 127.20.151.62:46755 was elected leader, sending a full tablet report...
I20260812 01:14:40.207165 12185 tablet_bootstrap.cc:492] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3: No bootstrap required, opened a new log
I20260812 01:14:40.211655 12181 raft_consensus.cc:348] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b688bc7851487a9e7d8f8251a71c4a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40963 } } peers { permanent_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44103 } } peers { permanent_uuid: "f3697307d96847079aebacef625372a3" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41141 } }
I20260812 01:14:40.212599 12181 raft_consensus.cc:374] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:40.213018 12181 raft_consensus.cc:729] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78239b47b824458ad4fbc2efd1c7fb8, State: Initialized, Role: FOLLOWER
I20260812 01:14:40.213224 12185 ts_tablet_manager.cc:1397] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20260812 01:14:40.218190 12181 consensus_queue.cc:260] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b688bc7851487a9e7d8f8251a71c4a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40963 } } peers { permanent_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44103 } } peers { permanent_uuid: "f3697307d96847079aebacef625372a3" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41141 } }
I20260812 01:14:40.224879 12185 raft_consensus.cc:348] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b688bc7851487a9e7d8f8251a71c4a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40963 } } peers { permanent_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44103 } } peers { permanent_uuid: "f3697307d96847079aebacef625372a3" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41141 } }
I20260812 01:14:40.225561 12185 raft_consensus.cc:374] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:40.225749 12185 raft_consensus.cc:729] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3697307d96847079aebacef625372a3, State: Initialized, Role: FOLLOWER
I20260812 01:14:40.226584 12185 consensus_queue.cc:260] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b688bc7851487a9e7d8f8251a71c4a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40963 } } peers { permanent_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44103 } } peers { permanent_uuid: "f3697307d96847079aebacef625372a3" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41141 } }
I20260812 01:14:40.234311 12181 ts_tablet_manager.cc:1428] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8: Time spent starting tablet: real 0.046s user 0.016s sys 0.001s
I20260812 01:14:40.234869 11674 heartbeater.cc:503] Master 127.20.151.62:46755 was elected leader, sending a full tablet report...
I20260812 01:14:40.235930 12185 ts_tablet_manager.cc:1428] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3: Time spent starting tablet: real 0.022s user 0.010s sys 0.011s
I20260812 01:14:40.239601 12136 heartbeater.cc:503] Master 127.20.151.62:46755 was elected leader, sending a full tablet report...
W20260812 01:14:40.264294 11904 tablet.cc:2369] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:14:40.311563 12189 raft_consensus.cc:482] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:14:40.311926 12189 raft_consensus.cc:504] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b688bc7851487a9e7d8f8251a71c4a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40963 } } peers { permanent_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44103 } } peers { permanent_uuid: "f3697307d96847079aebacef625372a3" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41141 } }
I20260812 01:14:40.314002 12189 leader_election.cc:288] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27b688bc7851487a9e7d8f8251a71c4a (127.20.151.2:40963), f3697307d96847079aebacef625372a3 (127.20.151.1:41141)
I20260812 01:14:40.321575 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.314370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:40963 (local address 127.20.151.3:44079)
0812 01:14:40.314920 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.314943 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:40.315074 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:40.315620 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:40.315625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:40.315656 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:40.315856 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:40.315868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.317790 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.317796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:40.319564 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:40.319577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.320807 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.320813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:40.321028 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.321054 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 01:14:40.321158 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 01:14:40.321294 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":3}
I20260812 01:14:40.321565 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.314887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:44079 (local address 127.20.151.2:40963)
0812 01:14:40.315110 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:40.315116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:40.315139 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:14:40.315217 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:40.315223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:40.315342 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:40.315436 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:40.316258 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.317360 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.319735 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.320622 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.320790 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.320866 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:40.321195 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 01:14:40.321324 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 01:14:40.322971 11838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4811318039b0411eaa98ca65673145a3" candidate_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27b688bc7851487a9e7d8f8251a71c4a" is_pre_election: true
I20260812 01:14:40.323249 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.315602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:48427 (local address 127.20.151.1:41141)
0812 01:14:40.316172 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:40.316178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:40.316345 (+ 167us) server_negotiation.cc:407] Connection header received
0812 01:14:40.316515 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:40.316521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:40.316611 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:40.316699 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:40.317458 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.318555 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.321089 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.322320 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.322470 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.322541 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:40.322859 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 01:14:40.322990 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406}
I20260812 01:14:40.323348 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.314639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41141 (local address 127.20.151.3:48427)
0812 01:14:40.316239 (+ 1600us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.316256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:40.316382 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:40.316860 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:40.316865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:40.316891 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:40.317255 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:40.317266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.318739 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.318744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:40.320879 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:40.320897 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.322521 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.322527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:40.322696 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.322727 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 01:14:40.322822 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 01:14:40.322950 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"mutex_wait_us":27,"thread_start_us":1289,"threads_started":1}
I20260812 01:14:40.323669 11838 raft_consensus.cc:2454] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78239b47b824458ad4fbc2efd1c7fb8 in term 0.
I20260812 01:14:40.324303 11597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4811318039b0411eaa98ca65673145a3" candidate_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3697307d96847079aebacef625372a3" is_pre_election: true
I20260812 01:14:40.324492 11945 leader_election.cc:302] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b688bc7851487a9e7d8f8251a71c4a, f78239b47b824458ad4fbc2efd1c7fb8; no voters:
I20260812 01:14:40.324891 12189 raft_consensus.cc:2781] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:14:40.324839 11597 raft_consensus.cc:2454] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78239b47b824458ad4fbc2efd1c7fb8 in term 0.
I20260812 01:14:40.325023 12189 raft_consensus.cc:482] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:14:40.325125 12189 raft_consensus.cc:3037] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:40.327564 12189 raft_consensus.cc:504] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b688bc7851487a9e7d8f8251a71c4a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40963 } } peers { permanent_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44103 } } peers { permanent_uuid: "f3697307d96847079aebacef625372a3" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41141 } }
I20260812 01:14:40.328182 12189 leader_election.cc:288] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [CANDIDATE]: Term 1 election: Requested vote from peers 27b688bc7851487a9e7d8f8251a71c4a (127.20.151.2:40963), f3697307d96847079aebacef625372a3 (127.20.151.1:41141)
I20260812 01:14:40.328651 11838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4811318039b0411eaa98ca65673145a3" candidate_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27b688bc7851487a9e7d8f8251a71c4a"
I20260812 01:14:40.328752 11597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4811318039b0411eaa98ca65673145a3" candidate_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3697307d96847079aebacef625372a3"
I20260812 01:14:40.328892 11838 raft_consensus.cc:3037] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:40.328953 11597 raft_consensus.cc:3037] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:40.332185 11838 raft_consensus.cc:2454] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f78239b47b824458ad4fbc2efd1c7fb8 in term 1.
I20260812 01:14:40.332185 11597 raft_consensus.cc:2454] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f78239b47b824458ad4fbc2efd1c7fb8 in term 1.
I20260812 01:14:40.332653 11945 leader_election.cc:302] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b688bc7851487a9e7d8f8251a71c4a, f78239b47b824458ad4fbc2efd1c7fb8; no voters:
I20260812 01:14:40.332976 12189 raft_consensus.cc:2781] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:14:40.333437 12189 raft_consensus.cc:686] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 1 LEADER]: Becoming Leader. State: Replica: f78239b47b824458ad4fbc2efd1c7fb8, State: Running, Role: LEADER
I20260812 01:14:40.333964 12189 consensus_queue.cc:237] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b688bc7851487a9e7d8f8251a71c4a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40963 } } peers { permanent_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44103 } } peers { permanent_uuid: "f3697307d96847079aebacef625372a3" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41141 } }
I20260812 01:14:40.339313 11344 catalog_manager.cc:5495] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f78239b47b824458ad4fbc2efd1c7fb8 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b688bc7851487a9e7d8f8251a71c4a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f78239b47b824458ad4fbc2efd1c7fb8" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 44103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3697307d96847079aebacef625372a3" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41141 } health_report { overall_health: UNKNOWN } } }
W20260812 01:14:40.343586 11678 tablet.cc:2369] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:14:40.383018 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.377265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.20.151.3:44103)
0812 01:14:40.377483 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:40.377488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:40.377506 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:14:40.377683 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:40.377687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:40.377776 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:40.377901 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:14:40.378242 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.379036 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.380632 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.381045 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.381189 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.381713 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:14:40.382389 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:14:40.382514 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:40.382654 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 01:14:40.382738 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 01:14:40.391767 11838 raft_consensus.cc:1261] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a [term 1 FOLLOWER]: Refusing update from remote peer f78239b47b824458ad4fbc2efd1c7fb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:14:40.391776 11597 raft_consensus.cc:1261] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3 [term 1 FOLLOWER]: Refusing update from remote peer f78239b47b824458ad4fbc2efd1c7fb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:14:40.392855 12189 consensus_queue.cc:1035] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3697307d96847079aebacef625372a3" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:14:40.393178 12203 consensus_queue.cc:1035] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27b688bc7851487a9e7d8f8251a71c4a" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 40963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:14:40.405046 12209 mvcc.cc:205] Tried to move back new op lower bound from 7317492860474347520 to 7317492860252360704. Current Snapshot: MvccSnapshot[applied={T|T < 7317492860474347520}]
I20260812 01:14:40.405876 12207 mvcc.cc:205] Tried to move back new op lower bound from 7317492860474347520 to 7317492860252360704. Current Snapshot: MvccSnapshot[applied={T|T < 7317492860474347520}]
I20260812 01:14:40.420352 11342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:14:40.420656 11342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:14:40.424813 11342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81c2deca005c4a38b91b338f67ca6aac: Sending DeleteTablet for 3 replicas of tablet 4811318039b0411eaa98ca65673145a3
I20260812 01:14:40.426549 11806 tablet_service.cc:1514] Processing DeleteTablet for tablet 4811318039b0411eaa98ca65673145a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:14:40 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:49742
I20260812 01:14:40.426578 11563 tablet_service.cc:1514] Processing DeleteTablet for tablet 4811318039b0411eaa98ca65673145a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:14:40 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:37438
I20260812 01:14:40.427372 12021 tablet_service.cc:1514] Processing DeleteTablet for tablet 4811318039b0411eaa98ca65673145a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:14:40 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:48536
I20260812 01:14:40.427572 12222 tablet_replica.cc:330] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: stopping tablet replica
I20260812 01:14:40.428040 12222 raft_consensus.cc:2229] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:14:40.428689 12222 raft_consensus.cc:2258] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:14:40.432595 12222 ts_tablet_manager.cc:1905] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:14:40.436751 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 11402
I20260812 01:14:40.437071 12226 tablet_replica.cc:330] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8: stopping tablet replica
I20260812 01:14:40.437767 12226 raft_consensus.cc:2229] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:14:40.438526 12226 raft_consensus.cc:2258] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:14:40.440521 12222 ts_tablet_manager.cc:1918] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:14:40.440743 12222 log.cc:1196] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/4811318039b0411eaa98ca65673145a3
I20260812 01:14:40.441008 12226 ts_tablet_manager.cc:1905] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:14:40.441371 12222 ts_tablet_manager.cc:1939] T 4811318039b0411eaa98ca65673145a3 P 27b688bc7851487a9e7d8f8251a71c4a: Deleting consensus metadata
I20260812 01:14:40.441617 12223 tablet_replica.cc:330] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3: stopping tablet replica
I20260812 01:14:40.442101 12223 raft_consensus.cc:2229] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:14:40.443540 11319 catalog_manager.cc:4841] TS 27b688bc7851487a9e7d8f8251a71c4a (127.20.151.2:40963): tablet 4811318039b0411eaa98ca65673145a3 (table test-table [id=3cdbd84dd03d4f20af678e6e670897cf]) successfully deleted
I20260812 01:14:40.449186 12226 ts_tablet_manager.cc:1918] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:14:40.449424 12226 log.cc:1196] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/4811318039b0411eaa98ca65673145a3
I20260812 01:14:40.450069 12226 ts_tablet_manager.cc:1939] T 4811318039b0411eaa98ca65673145a3 P f78239b47b824458ad4fbc2efd1c7fb8: Deleting consensus metadata
I20260812 01:14:40.451833 12223 raft_consensus.cc:2258] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:14:40.453655 11320 catalog_manager.cc:4841] TS f78239b47b824458ad4fbc2efd1c7fb8 (127.20.151.3:44103): tablet 4811318039b0411eaa98ca65673145a3 (table test-table [id=3cdbd84dd03d4f20af678e6e670897cf]) successfully deleted
I20260812 01:14:40.454667 12223 ts_tablet_manager.cc:1905] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:14:40.460301 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.439604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.20.151.1:41141)
0812 01:14:40.439823 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:40.439830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:40.440641 (+ 811us) server_negotiation.cc:407] Connection header received
0812 01:14:40.440722 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:40.440727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:40.440813 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:40.440910 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:40.443711 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.444905 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.447165 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.447884 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.448033 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.451913 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:40.451933 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:40.451953 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:40.452002 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:40.456165 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:40.457306 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:40.457317 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:40.457321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:40.457536 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:40.458032 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:40.458037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:40.458041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:40.458571 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:40.458933 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:40.459338 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 01:14:40.459619 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 01:14:40.461043 11663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:40.560235 12223 ts_tablet_manager.cc:1918] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:14:40.560523 12223 log.cc:1196] T 4811318039b0411eaa98ca65673145a3 P f3697307d96847079aebacef625372a3: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/4811318039b0411eaa98ca65673145a3
I20260812 01:14:40.560897 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 11402
I20260812 01:14:40.585691 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 11687
W20260812 01:14:40.585755 11315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.151.1:41141 (error 108)
I20260812 01:14:40.587850 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.587341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41141 (local address 127.0.0.1:37462)
0812 01:14:40.587534 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.587677 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 01:14:40.588116 11315 catalog_manager.cc:4584] TS f3697307d96847079aebacef625372a3 (127.20.151.1:41141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4811318039b0411eaa98ca65673145a3: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41141: connect: Connection refused (error 111)
I20260812 01:14:40.600237 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.586916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.20.151.2:40963)
0812 01:14:40.587133 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:40.587139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:40.587158 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:14:40.587227 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:40.587232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:40.587337 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:40.587437 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:40.589013 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.589766 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.591057 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.591501 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.591630 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.594211 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:40.594227 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:40.594243 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:40.594282 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:40.597302 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:40.598031 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:40.598039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:40.598041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:40.598209 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:40.598601 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:40.598605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:40.598607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:40.598988 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:40.599302 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:40.599656 (+ 354us) server_negotiation.cc:299] Negotiation successful
0812 01:14:40.599906 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 01:14:40.600970 11897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:40.642023 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.641420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41141 (local address 127.0.0.1:37474)
0812 01:14:40.641691 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.641823 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260812 01:14:40.643322 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.642854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41141 (local address 127.0.0.1:37476)
0812 01:14:40.643030 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.643127 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 01:14:40.692346 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 11687
I20260812 01:14:40.704083 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.703518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41141 (local address 127.0.0.1:37490)
0812 01:14:40.703787 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.703880 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260812 01:14:40.705412 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.704996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41141 (local address 127.0.0.1:37506)
0812 01:14:40.705194 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.705268 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 01:14:40.730423 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 11906
I20260812 01:14:40.746254 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.731342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.20.151.3:44103)
0812 01:14:40.731550 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:40.731557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:40.731576 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:14:40.731741 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:40.731745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:40.731850 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:40.731951 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:40.733310 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.734142 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.736137 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:40.736575 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:40.736678 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:40.740033 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:40.740045 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:40.740064 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:40.740105 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:40.743185 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:40.744116 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:40.744123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:40.744125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:40.744276 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:40.744700 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:40.744703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:40.744705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:40.745067 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:40.745328 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:40.745722 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 01:14:40.745905 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 01:14:40.747001 12124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:40.786231 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.785644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41141 (local address 127.0.0.1:37518)
0812 01:14:40.785906 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.786010 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260812 01:14:40.787420 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:40.786959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41141 (local address 127.0.0.1:37526)
0812 01:14:40.787167 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:40.787242 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260812 01:14:40.876642 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 11906
I20260812 01:14:40.900647 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 11293
I20260812 01:14:40.902004 11394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:41.019555 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:41.018564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41141 (local address 127.0.0.1:37532)
0812 01:14:41.018775 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:41.018880 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 01:14:41.019748 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 11293
I20260812 01:14:41.053747 21084 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 01:14:41.400177 21084 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 01:14:41.400390 21084 mini_ranger.cc:66] Stopping Ranger...
I20260812 01:14:41.757150 21084 mini_ranger.cc:68] Stopped Ranger
2026-08-12 01:14:41.757 UTC [9603] LOG: received smart shutdown request
2026-08-12 01:14:41.760 UTC [9603] LOG: background worker "logical replication launcher" (PID 9616) exited with exit code 1
2026-08-12 01:14:41.760 UTC [9607] LOG: shutting down
2026-08-12 01:14:41.936 UTC [9603] LOG: database system is shut down
2026-08-12T01:14:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55570 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:14:42 dist-test-slave-21b9 krb5kdc[12254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:14:42 dist-test-slave-21b9 krb5kdc[12254](info): set up 2 sockets
Aug 12 01:14:42 dist-test-slave-21b9 krb5kdc[12254](info): commencing operation
krb5kdc: starting...
W20260812 01:14:43.331251 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:14:43 dist-test-slave-21b9 krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:14:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:14:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:14:43.660351 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43393
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:43393,127.20.151.61:44889,127.20.151.60:45157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:14:44.017107 12276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:44.017565 12276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:44.017736 12276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:44.025557 12276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:14:44.025676 12276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:44.025732 12276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:44.025790 12276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:14:44.025835 12276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:14:44.036434 12276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:43393,127.20.151.61:44889,127.20.151.60:45157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:44.037798 12276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:44.039150 12276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:44.051651 12284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:44.051604 12285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:44.051568 12287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:44.052886 12276 server_base.cc:1034] running on GCE node
I20260812 01:14:44.053803 12276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:44.055428 12276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:44.056654 12276 hybrid_clock.cc:648] HybridClock initialized: now 1786497284056607 us; error 49 us; skew 500 ppm
I20260812 01:14:44.057024 12276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:44 dist-test-slave-21b9 krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497284, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:44.064777 12276 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:14:44.066535 12276 webserver.cc:460] Webserver started at http://127.20.151.62:33455/ using document root <none> and password file <none>
I20260812 01:14:44.067313 12276 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:44.067449 12276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:44.067744 12276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:14:44.070927 12276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "0414ba195320412597531b5514359017"
format_stamp: "Formatted at 2026-08-12 01:14:44 on dist-test-slave-21b9"
server_key: "4267c0264d213007a16921416aa1383d"
server_key_iv: "00ec49929930c6049757124feb0092bf"
server_key_version: "encryptionkey@0"
I20260812 01:14:44.071842 12276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "0414ba195320412597531b5514359017"
format_stamp: "Formatted at 2026-08-12 01:14:44 on dist-test-slave-21b9"
server_key: "4267c0264d213007a16921416aa1383d"
server_key_iv: "00ec49929930c6049757124feb0092bf"
server_key_version: "encryptionkey@0"
I20260812 01:14:44.077332 12276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260812 01:14:44.081087 12298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:44.082460 12276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:14:44.082687 12276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "0414ba195320412597531b5514359017"
format_stamp: "Formatted at 2026-08-12 01:14:44 on dist-test-slave-21b9"
server_key: "4267c0264d213007a16921416aa1383d"
server_key_iv: "00ec49929930c6049757124feb0092bf"
server_key_version: "encryptionkey@0"
I20260812 01:14:44.082890 12276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:44.099195 12276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:44.104206 12276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:44.104560 12276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:44.123567 12276 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:43393
I20260812 01:14:44.123569 12383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:43393 every 8 connection(s)
I20260812 01:14:44.125094 12276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:14:44.128974 12384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:44.134034 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 12276
I20260812 01:14:44.134264 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:14:44.134896 21084 external_mini_cluster.cc:1428] Setting key 684dea0c670b1a2d8b430b6b408b1217
I20260812 01:14:44.136405 12384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:14:44 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497284, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:44.161369 12390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.139983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.20.151.62:43393)
0812 01:14:44.141848 (+ 1865us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:44.141857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:44.141915 (+ 58us) server_negotiation.cc:407] Connection header received
0812 01:14:44.143656 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:44.143663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:44.143924 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:44.144126 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:44.145553 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.146837 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.148360 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.149272 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.149497 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:44.153526 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:44.153536 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:44.153548 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:44.153582 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:44.156601 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:44.157364 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:44.157373 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:44.157377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:44.157552 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:44.157997 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:44.158000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:44.158002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:44.158565 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:44.158785 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:44.159446 (+ 661us) server_negotiation.cc:299] Negotiation successful
0812 01:14:44.159762 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1626,"spinlock_wait_cycles":27392,"thread_start_us":1371,"threads_started":1}
I20260812 01:14:44.161412 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.139991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:43393 (local address 127.0.0.1:38706)
0812 01:14:44.140967 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.141026 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:44.143209 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:44.144177 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:44.144184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:44.144956 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:44.145328 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:44.145347 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.147018 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.147022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:44.148203 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:44.148217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.149589 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.149596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:44.149730 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:44.150641 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:44.150667 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:44.153252 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:44.156754 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:44.156767 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:44.156782 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:44.157215 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:44.157677 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:44.157683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:44.157688 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:44.157867 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:44.158725 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:44.158733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:44.158996 (+ 263us) client_negotiation.cc:769] Sending connection context
0812 01:14:44.159246 (+ 250us) client_negotiation.cc:240] Negotiation successful
0812 01:14:44.159950 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":143,"thread_start_us":417,"threads_started":1}
WARNING: no policy specified for kudu/127.20.151.61@KRBTEST.COM; defaulting to no policy
I20260812 01:14:44.164855 12384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:14:44.165920 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.165475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36532)
0812 01:14:44.165669 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.165789 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260812 01:14:44.166174 12302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.61@KRBTEST.COM" created.
I20260812 01:14:44.167909 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.167508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36544)
0812 01:14:44.167683 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.167760 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260812 01:14:44.168133 12384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:44889: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab.
Entry for principal kudu/127.20.151.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab.
I20260812 01:14:44.203564 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.61:44889
--webserver_interface=127.20.151.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab
--principal=kudu/127.20.151.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:43393,127.20.151.61:44889,127.20.151.60:45157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:14:44.217578 12384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } attempt: 1
I20260812 01:14:44.218928 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.218375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36548)
0812 01:14:44.218608 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.218750 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 01:14:44.220047 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.219627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36560)
0812 01:14:44.219800 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.219881 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260812 01:14:44.220274 12384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:44889: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
I20260812 01:14:44.288599 12384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } attempt: 2
I20260812 01:14:44.289991 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.289377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36572)
0812 01:14:44.289668 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.289791 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 01:14:44.291121 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.290662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36578)
0812 01:14:44.290829 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.290933 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 01:14:44.291357 12384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:44889: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
I20260812 01:14:44.382623 12384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } attempt: 3
I20260812 01:14:44.383983 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.383435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36580)
0812 01:14:44.383670 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.383803 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 01:14:44.385093 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.384665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36588)
0812 01:14:44.384820 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.384906 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 01:14:44.385313 12384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:44889: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
I20260812 01:14:44.528612 12384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } attempt: 4
I20260812 01:14:44.530119 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.529422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36592)
0812 01:14:44.529787 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.529909 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260812 01:14:44.531185 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.530766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36598)
0812 01:14:44.530916 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.531036 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 01:14:44.531407 12384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:44889: Network error: Client connection negotiation failed: client connection to 127.20.151.61:44889: connect: Connection refused (error 111)
W20260812 01:14:44.558887 12398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:44.559209 12398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:44.559369 12398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:44.566784 12398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:14:44.566901 12398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:44.566955 12398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:44.567039 12398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:14:44.567098 12398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:14:44.577276 12398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:43393,127.20.151.61:44889,127.20.151.60:45157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab
--principal=kudu/127.20.151.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.61:44889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.151.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:44.578687 12398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:44.580157 12398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:44.592192 12404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:44.592254 12403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:44.592319 12407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:44.593163 12398 server_base.cc:1034] running on GCE node
I20260812 01:14:44.593961 12398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:44.595449 12398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:44.596683 12398 hybrid_clock.cc:648] HybridClock initialized: now 1786497284596619 us; error 54 us; skew 500 ppm
I20260812 01:14:44.597047 12398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:44 dist-test-slave-21b9 krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497284, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:44.600309 12398 init.cc:377] Logged in from keytab as kudu/127.20.151.61@KRBTEST.COM (short username kudu)
I20260812 01:14:44.601969 12398 webserver.cc:460] Webserver started at http://127.20.151.61:44443/ using document root <none> and password file <none>
I20260812 01:14:44.602686 12398 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:44.602810 12398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:44.603122 12398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:14:44.606237 12398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/data/instance:
uuid: "b6fbcf6ffca9433a986a4388e742a318"
format_stamp: "Formatted at 2026-08-12 01:14:44 on dist-test-slave-21b9"
server_key: "babbe3eac50b4009ee5d032d8c159134"
server_key_iv: "07478ed0e5d7ddcebbebce703644ade5"
server_key_version: "encryptionkey@0"
I20260812 01:14:44.607059 12398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/wal/instance:
uuid: "b6fbcf6ffca9433a986a4388e742a318"
format_stamp: "Formatted at 2026-08-12 01:14:44 on dist-test-slave-21b9"
server_key: "babbe3eac50b4009ee5d032d8c159134"
server_key_iv: "07478ed0e5d7ddcebbebce703644ade5"
server_key_version: "encryptionkey@0"
I20260812 01:14:44.612582 12398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 01:14:44.616406 12421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:44.617882 12398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260812 01:14:44.618067 12398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/wal
uuid: "b6fbcf6ffca9433a986a4388e742a318"
format_stamp: "Formatted at 2026-08-12 01:14:44 on dist-test-slave-21b9"
server_key: "babbe3eac50b4009ee5d032d8c159134"
server_key_iv: "07478ed0e5d7ddcebbebce703644ade5"
server_key_version: "encryptionkey@0"
I20260812 01:14:44.618273 12398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:44.642232 12398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:44.646654 12398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:44.646960 12398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:44.666482 12398 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.61:44889
I20260812 01:14:44.666527 12509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.61:44889 every 8 connection(s)
I20260812 01:14:44.667968 12398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
I20260812 01:14:44.671877 12511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:44.676608 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 12398
I20260812 01:14:44.676828 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-1/wal/instance
I20260812 01:14:44.677413 21084 external_mini_cluster.cc:1428] Setting key 9091c9c0ef216a23c4772907a63fbb1e
I20260812 01:14:44.679960 12511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:14:44 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497284, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:44.704787 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.683454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.20.151.62:43393)
0812 01:14:44.684124 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:44.684131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:44.685281 (+ 1150us) server_negotiation.cc:407] Connection header received
0812 01:14:44.687846 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:44.687851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:44.687927 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:44.688019 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:44.690174 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.690946 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.692957 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.693625 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.693760 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:44.697850 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:44.697856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:44.697860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:44.697909 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:44.700866 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:44.702010 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:44.702015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:44.702018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:44.702214 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:44.702763 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:44.702767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:44.702770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:44.703102 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:44.703225 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:44.704238 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0812 01:14:44.704448 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":390,"threads_started":1}
I20260812 01:14:44.706239 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.684176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:43393 (local address 127.0.0.1:38712)
0812 01:14:44.685079 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.685135 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:44.687462 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:44.688322 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:44.688334 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:44.689102 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:44.689949 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:44.689974 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.691128 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.691133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:44.692779 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:44.692793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.693885 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.693891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:44.694120 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:44.694913 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:44.694947 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:44.697557 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:44.701070 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:44.701082 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:44.701105 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:44.701829 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:44.702348 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:44.702352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:44.702355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:44.702624 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:44.703250 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:44.703258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:44.703786 (+ 528us) client_negotiation.cc:769] Sending connection context
0812 01:14:44.704109 (+ 323us) client_negotiation.cc:240] Negotiation successful
0812 01:14:44.704709 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":350,"threads_started":1}
I20260812 01:14:44.708227 12511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.151.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.60@KRBTEST.COM" created.
Aug 12 01:14:44 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497284, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
I20260812 01:14:44.724270 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.708931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36610)
0812 01:14:44.709120 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.709139 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:44.709269 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:44.710451 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:44.710456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:44.710809 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:44.710966 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:44.710975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.712477 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.712481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:44.713625 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:44.713641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.714806 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.714811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:44.714988 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:44.715622 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:44.715649 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:44.717736 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:44.721723 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:44.721728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:44.721730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:44.722242 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:44.722624 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:44.722628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:44.722630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:44.722778 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:44.723505 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:44.723509 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:44.723645 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 01:14:44.723773 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 01:14:44.723996 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 01:14:44.724619 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.709220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.20.151.61:44889)
0812 01:14:44.709990 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:44.709997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:44.710023 (+ 26us) server_negotiation.cc:407] Connection header received
0812 01:14:44.710070 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:44.710075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:44.710265 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:44.710385 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:44.711115 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.712342 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.713794 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.714622 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.714750 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:44.717984 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:44.718000 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:44.718020 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:44.718057 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:44.721568 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:44.722372 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:44.722382 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:44.722385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:44.722530 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:44.722884 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:44.722888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:44.722891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:44.723375 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:44.723597 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:44.723945 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 01:14:44.724222 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":469,"threads_started":1}
I20260812 01:14:44.727046 12511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:14:44.728400 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.727844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48836)
0812 01:14:44.728057 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.728212 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260812 01:14:44.728749 12426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
I20260812 01:14:44.730353 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.730015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48840)
0812 01:14:44.730159 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.730240 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 01:14:44.730633 12511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:45157: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab.
Entry for principal kudu/127.20.151.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab.
I20260812 01:14:44.754565 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.60:45157
--webserver_interface=127.20.151.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab
--principal=kudu/127.20.151.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:43393,127.20.151.61:44889,127.20.151.60:45157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:14:44.779953 12511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } attempt: 1
I20260812 01:14:44.781121 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.780637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48852)
0812 01:14:44.780839 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.780956 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 01:14:44.782248 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.781913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48856)
0812 01:14:44.782066 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.782138 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 01:14:44.782474 12511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:45157: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
I20260812 01:14:44.830683 12384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } attempt: 5
Aug 12 01:14:44 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497284, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
I20260812 01:14:44.843984 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.831552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.20.151.61:44889)
0812 01:14:44.831727 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:44.831733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:44.831774 (+ 41us) server_negotiation.cc:407] Connection header received
0812 01:14:44.832004 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:44.832008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:44.832090 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:44.832178 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:44.833025 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.833725 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.834880 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.835438 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.835550 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:44.838696 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:44.838702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:44.838705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:44.838738 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:44.841368 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:44.842027 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:44.842032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:44.842035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:44.842159 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:44.842536 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:44.842539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:44.842541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:44.842849 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:44.842990 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:44.843421 (+ 431us) server_negotiation.cc:299] Negotiation successful
0812 01:14:44.843638 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 01:14:44.843971 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.831472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36616)
0812 01:14:44.831690 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.831712 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:44.831864 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:44.832238 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:44.832243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:44.832646 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:44.832855 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:44.832865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.833862 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.833866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:44.834735 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:44.834746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:44.835583 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:44.835588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:44.835777 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:44.836457 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:44.836477 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:44.838508 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:44.841530 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:44.841535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:44.841540 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:44.841905 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:44.842272 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:44.842277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:44.842281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:44.842428 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:44.842971 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:44.842976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:44.843130 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 01:14:44.843259 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 01:14:44.843624 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 01:14:44.845309 12384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:14:44.846318 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.845930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48870)
0812 01:14:44.846102 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.846177 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 01:14:44.847476 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.847120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48882)
0812 01:14:44.847260 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.847376 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 01:14:44.847666 12384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:45157: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
I20260812 01:14:44.850673 12511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } attempt: 2
I20260812 01:14:44.851619 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.851191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48886)
0812 01:14:44.851384 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.851472 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 01:14:44.852566 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.852211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48894)
0812 01:14:44.852354 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.852456 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 01:14:44.852764 12511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:45157: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
I20260812 01:14:44.898914 12384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } attempt: 1
I20260812 01:14:44.900216 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.899671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48910)
0812 01:14:44.899894 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.899987 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 01:14:44.901317 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.900989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48920)
0812 01:14:44.901117 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.901205 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 01:14:44.901553 12384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:45157: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
I20260812 01:14:44.943982 12511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } attempt: 3
I20260812 01:14:44.945196 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.944676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48934)
0812 01:14:44.944916 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.945013 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 01:14:44.946553 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.946008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48946)
0812 01:14:44.946298 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.946384 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
W20260812 01:14:44.946803 12511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:45157: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
I20260812 01:14:44.969774 12384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } attempt: 2
I20260812 01:14:44.971087 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.970571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48962)
0812 01:14:44.970820 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.970910 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 01:14:44.972151 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:44.971780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48972)
0812 01:14:44.971956 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:44.972028 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 01:14:44.972391 12384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:45157: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
I20260812 01:14:45.078691 12384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } attempt: 3
I20260812 01:14:45.079972 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.079489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48980)
0812 01:14:45.079707 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:45.079800 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 01:14:45.081159 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.080737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48982)
0812 01:14:45.080952 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:45.081036 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260812 01:14:45.081399 12384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:45157: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
I20260812 01:14:45.090040 12511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } attempt: 4
I20260812 01:14:45.091238 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.090740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:48996)
0812 01:14:45.090982 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:45.091081 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 01:14:45.092259 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.091927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:49000)
0812 01:14:45.092082 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:45.092153 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 01:14:45.092509 12511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:45157: Network error: Client connection negotiation failed: client connection to 127.20.151.60:45157: connect: Connection refused (error 111)
W20260812 01:14:45.110522 12527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:45.110834 12527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:45.110975 12527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:45.118461 12527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:14:45.118589 12527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:45.118649 12527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:45.118734 12527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:14:45.118793 12527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:14:45.129374 12527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:43393,127.20.151.61:44889,127.20.151.60:45157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab
--principal=kudu/127.20.151.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.60:45157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.151.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:45.130750 12527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:45.132079 12527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:45.144307 12540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:45.144359 12537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:45.144269 12536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:45.145252 12527 server_base.cc:1034] running on GCE node
I20260812 01:14:45.146062 12527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:45.147588 12527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:45.148818 12527 hybrid_clock.cc:648] HybridClock initialized: now 1786497285148754 us; error 43 us; skew 500 ppm
I20260812 01:14:45.149184 12527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:45 dist-test-slave-21b9 krb5kdc[12254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497285, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:45.152525 12527 init.cc:377] Logged in from keytab as kudu/127.20.151.60@KRBTEST.COM (short username kudu)
I20260812 01:14:45.154032 12527 webserver.cc:460] Webserver started at http://127.20.151.60:42357/ using document root <none> and password file <none>
I20260812 01:14:45.154743 12527 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:45.154846 12527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:45.155150 12527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:14:45.158253 12527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/data/instance:
uuid: "2b8ffa90162d4153ae83666c44cb7db8"
format_stamp: "Formatted at 2026-08-12 01:14:45 on dist-test-slave-21b9"
server_key: "954a7c5686f252785c68727e99b5911b"
server_key_iv: "cb8b60780652a2ede9c3930d806754a6"
server_key_version: "encryptionkey@0"
I20260812 01:14:45.159027 12527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/wal/instance:
uuid: "2b8ffa90162d4153ae83666c44cb7db8"
format_stamp: "Formatted at 2026-08-12 01:14:45 on dist-test-slave-21b9"
server_key: "954a7c5686f252785c68727e99b5911b"
server_key_iv: "cb8b60780652a2ede9c3930d806754a6"
server_key_version: "encryptionkey@0"
I20260812 01:14:45.164341 12527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:14:45.167891 12550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:45.169345 12527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:14:45.169557 12527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/wal
uuid: "2b8ffa90162d4153ae83666c44cb7db8"
format_stamp: "Formatted at 2026-08-12 01:14:45 on dist-test-slave-21b9"
server_key: "954a7c5686f252785c68727e99b5911b"
server_key_iv: "cb8b60780652a2ede9c3930d806754a6"
server_key_version: "encryptionkey@0"
I20260812 01:14:45.169780 12527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:45.187008 12527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:45.191406 12527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:45.191715 12527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:45.211303 12527 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.60:45157
I20260812 01:14:45.211349 12646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.60:45157 every 8 connection(s)
I20260812 01:14:45.212759 12527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
I20260812 01:14:45.216586 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 12527
I20260812 01:14:45.216706 12647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:45.216795 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786497066589499-21084-0/minicluster-data/master-2/wal/instance
I20260812 01:14:45.217319 21084 external_mini_cluster.cc:1428] Setting key bf60567cacd8785276425854b39fbb31
I20260812 01:14:45.227159 12647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:14:45 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
Aug 12 01:14:45 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497285, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:45.242928 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.219435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.20.151.60:45157)
0812 01:14:45.220420 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:45.220436 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:45.220489 (+ 53us) server_negotiation.cc:407] Connection header received
0812 01:14:45.222192 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:45.222225 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:45.222769 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:45.223248 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:45.224704 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.226639 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.228181 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.228904 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.229093 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.233840 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:45.233851 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:45.233862 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:45.233900 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:45.237583 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.238605 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.238614 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.238624 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.238849 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.239387 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.239391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.239393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.240175 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:45.240485 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:45.240999 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 01:14:45.241486 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":444,"threads_started":1}
I20260812 01:14:45.246976 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.229757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:43393 (local address 127.0.0.1:38718)
0812 01:14:45.230631 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:45.230666 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:45.230822 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:45.231215 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:45.231222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:45.231878 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:45.232204 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:45.232218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.233328 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.233334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:45.234633 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:45.234650 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.235422 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.235426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:45.235623 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.236305 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:45.236324 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:45.239385 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:45.244248 (+ 4863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:45.244260 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:45.244267 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:45.244725 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:45.245127 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:45.245130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:45.245133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:45.245316 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:45.245926 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:45.245933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:45.246142 (+ 209us) client_negotiation.cc:769] Sending connection context
0812 01:14:45.246348 (+ 206us) client_negotiation.cc:240] Negotiation successful
0812 01:14:45.246700 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":531,"threads_started":1}
I20260812 01:14:45.248045 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.229782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.20.151.62:43393)
0812 01:14:45.230284 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:45.230290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:45.230763 (+ 473us) server_negotiation.cc:407] Connection header received
0812 01:14:45.230965 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:45.230969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:45.231033 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:45.231125 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:45.232371 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.233154 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.234824 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.235236 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.235387 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.239621 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:45.239629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:45.239634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:45.239687 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:45.243079 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.244853 (+ 1774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.244859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.244863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.245003 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.245439 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.245444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.245448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.245783 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:45.245933 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:45.247594 (+ 1661us) server_negotiation.cc:299] Negotiation successful
0812 01:14:45.247766 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":268,"threads_started":1}
I20260812 01:14:45.249408 12647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:14:45 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
Aug 12 01:14:45 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497285, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
I20260812 01:14:45.258617 12384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } attempt: 4
I20260812 01:14:45.261416 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.246055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.20.151.62:43393)
0812 01:14:45.246659 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:45.246663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:45.246681 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:14:45.246719 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:45.246722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:45.246772 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:45.246871 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:45.248436 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.249391 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.250896 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.251394 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.251506 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.255611 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:45.255617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:45.255623 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:45.255664 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:45.258313 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.259207 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.259212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.259216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.259396 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.259835 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.259840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.259843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.260216 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:45.260469 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:45.260807 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 01:14:45.261022 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":288,"threads_started":1}
I20260812 01:14:45.263224 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.250127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:44889 (local address 127.0.0.1:36630)
0812 01:14:45.250284 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:45.250301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:45.250385 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:45.250851 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:45.250855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:45.251236 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:45.251432 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:45.251440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.252372 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.252378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:45.253261 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:45.253273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.254003 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.254008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:45.254234 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.254751 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:45.254767 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:45.256865 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:45.260415 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:45.260419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:45.260423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:45.260932 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:45.261404 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:45.261409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:45.261412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:45.261603 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:45.262288 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:45.262291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:45.262436 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 01:14:45.262579 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 01:14:45.262869 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 01:14:45.263315 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.250365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.20.151.61:44889)
0812 01:14:45.250555 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:45.250560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:45.250607 (+ 47us) server_negotiation.cc:407] Connection header received
0812 01:14:45.250655 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:45.250658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:45.250716 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:45.250799 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:45.251583 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.252230 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.253399 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.253853 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.253961 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.257067 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:45.257074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:45.257079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:45.257119 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:45.260260 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.261074 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.261080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.261083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.261241 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.261759 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.261764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.261768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.262116 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:45.262213 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:45.262718 (+ 505us) server_negotiation.cc:299] Negotiation successful
0812 01:14:45.262937 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:14:45.264258 12647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:14:45 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497284, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
Aug 12 01:14:45 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497285, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
Aug 12 01:14:45 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
I20260812 01:14:45.277269 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.259162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:49022)
0812 01:14:45.259454 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:45.259472 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:45.259589 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:45.259974 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:45.259979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:45.260460 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:45.260676 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:45.260688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.262017 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.262022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:45.263193 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:45.263207 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.264729 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.264736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:45.264941 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.265522 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:45.265546 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:45.268354 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:45.272487 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:45.272493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:45.272497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:45.273017 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:45.273588 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:45.273594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:45.273597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:45.275457 (+ 1860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:45.276234 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:45.276239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:45.276425 (+ 186us) client_negotiation.cc:769] Sending connection context
0812 01:14:45.276557 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 01:14:45.276861 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260812 01:14:45.278281 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.259335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.20.151.60:45157)
0812 01:14:45.259490 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:45.259495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:45.259514 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:14:45.259705 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:45.259710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:45.259776 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:45.259918 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:45.260848 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.261855 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.263790 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.264544 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.265218 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.268590 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:45.268597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:45.268602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:45.268643 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:45.272329 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.273268 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.273274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.273277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.273442 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.275609 (+ 2167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.275614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.275622 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.276080 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:45.277556 (+ 1476us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:45.277689 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 01:14:45.277935 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 01:14:45.281581 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.265092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:49024)
0812 01:14:45.265274 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:45.265291 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:45.265378 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:45.266315 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:45.266321 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:45.266850 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:45.267064 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:45.267087 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.268876 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.268882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:45.270444 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:45.270462 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.271602 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.271609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:45.271817 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.272364 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:45.272387 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:45.275303 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:45.278892 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:45.278897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:45.278901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:45.279457 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:45.279838 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:45.279844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:45.279847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:45.279989 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:45.280558 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:45.280564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:45.280719 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 01:14:45.280878 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 01:14:45.281214 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 01:14:45.281848 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.265319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.20.151.60:45157)
0812 01:14:45.265982 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:45.265987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:45.266009 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:14:45.266063 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:45.266067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:45.266140 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:45.266245 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:45.267254 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.268717 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.270617 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.271427 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.272125 (+ 698us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.275531 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:45.275539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:45.275545 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:45.275585 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:45.278743 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.279586 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.279591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.279594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.279730 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.280079 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.280084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.280088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.280409 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:45.280585 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:45.281029 (+ 444us) server_negotiation.cc:299] Negotiation successful
0812 01:14:45.281373 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":544,"thread_start_us":321,"threads_started":1}
I20260812 01:14:45.284307 12384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017: Bootstrap starting.
I20260812 01:14:45.285645 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.264062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.20.151.61:44889)
0812 01:14:45.267013 (+ 2951us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:45.267019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:45.267036 (+ 17us) server_negotiation.cc:407] Connection header received
0812 01:14:45.267082 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:45.267087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:45.267154 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:45.267245 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:45.269040 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.270114 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.271909 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.272528 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.272632 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.278251 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:45.278258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:45.278263 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:45.278305 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:45.281739 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.283197 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.283203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.283206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.283404 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.283816 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.283822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.283826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.284130 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:45.284430 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:45.284976 (+ 546us) server_negotiation.cc:299] Negotiation successful
0812 01:14:45.285207 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2833,"thread_start_us":12463,"threads_started":1}
I20260812 01:14:45.287835 12647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8: Bootstrap starting.
I20260812 01:14:45.288887 12384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:45.290320 12384 log.cc:824] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:45.290601 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:14:45.291726 12647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:45.293025 12647 log.cc:824] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:45.293946 12384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017: No bootstrap required, opened a new log
I20260812 01:14:45.303061 12647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8: No bootstrap required, opened a new log
I20260812 01:14:45.304943 12384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } }
I20260812 01:14:45.305516 12384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:45.305709 12384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0414ba195320412597531b5514359017, State: Initialized, Role: FOLLOWER
I20260812 01:14:45.306499 12384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } }
I20260812 01:14:45.310520 12677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } } }
I20260812 01:14:45.312897 12677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:14:45.312776 12384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:14:45.314785 12647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } }
I20260812 01:14:45.315459 12647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:45.315933 12647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b8ffa90162d4153ae83666c44cb7db8, State: Initialized, Role: FOLLOWER
I20260812 01:14:45.316848 12647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } }
I20260812 01:14:45.321187 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } } }
I20260812 01:14:45.321655 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:14:45.322970 12647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:14:45.338189 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.306888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.20.151.61:44889)
0812 01:14:45.307086 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:45.307093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:45.314817 (+ 7724us) server_negotiation.cc:407] Connection header received
0812 01:14:45.315746 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:45.315753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:45.315835 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:45.315902 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:45.320394 (+ 4492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.321483 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.323992 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.324739 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.324849 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.329396 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:45.329404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:45.329409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:45.329460 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:45.333234 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.334350 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.334357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.334361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.334591 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.335304 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.335310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.335314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.335744 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:45.336068 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:45.337494 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0812 01:14:45.337780 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 01:14:45.338289 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.303460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.20.151.62:43393)
0812 01:14:45.303724 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:45.303730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:45.306687 (+ 2957us) server_negotiation.cc:407] Connection header received
0812 01:14:45.308706 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:45.308714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:45.308814 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:45.308891 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:45.314534 (+ 5643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.315663 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.317730 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.318347 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.318439 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.325009 (+ 6570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:45.325017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:45.325023 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:45.325075 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:45.330515 (+ 5440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.333652 (+ 3137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.333659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.333663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.333857 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.335230 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.335237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.335241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.335824 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:45.336143 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:45.337626 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0812 01:14:45.337903 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 01:14:45.343115 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.302950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.20.151.60:45157)
0812 01:14:45.303168 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:45.303175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:45.308347 (+ 5172us) server_negotiation.cc:407] Connection header received
0812 01:14:45.311834 (+ 3487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:45.311841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:45.311937 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:45.312011 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:45.315038 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.316069 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.318176 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.318813 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.318909 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.326043 (+ 7134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:45.326051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:45.326056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:45.326103 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:45.332459 (+ 6356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.336794 (+ 4335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.336801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.336805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.336997 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.340465 (+ 3468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.340472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.340476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.340947 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:45.341252 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:45.342466 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0812 01:14:45.342750 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 01:14:45.355157 12719 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:14:45.355424 12719 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 01:14:45.357692 12307 tablet.cc:2369] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:14:45.363672 12527 master_runner.cc:186] Error getting master registration for 127.20.151.61:44889: OK, instance_id { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" instance_seqno: 1786497284660892 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:14:45.375945 12527 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 01:14:45.356841 12276 master_runner.cc:186] Error getting master registration for 127.20.151.61:44889: OK, instance_id { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" instance_seqno: 1786497284660892 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:14:45.376740 12276 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 01:14:45.378729 12714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:14:45.378857 12714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:14:45.391765 12511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } attempt: 5
Aug 12 01:14:45 dist-test-slave-21b9 krb5kdc[12254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497284, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
I20260812 01:14:45.408063 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.392489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:45157 (local address 127.0.0.1:49046)
0812 01:14:45.392797 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:45.392834 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:45.392969 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:45.393697 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:45.393703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:45.394323 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:45.394551 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:45.394566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.396920 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.396924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:45.397834 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:45.397848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.398625 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.398631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:45.398855 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.399484 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:45.399545 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:45.401694 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:45.405215 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:45.405221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:45.405225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:45.405683 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:45.406082 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:45.406088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:45.406091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:45.406232 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:45.406997 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:45.407001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:45.407171 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 01:14:45.407355 (+ 184us) client_negotiation.cc:240] Negotiation successful
0812 01:14:45.407683 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260812 01:14:45.409256 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:45.392582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.20.151.60:45157)
0812 01:14:45.392862 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:45.392867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:45.392891 (+ 24us) server_negotiation.cc:407] Connection header received
0812 01:14:45.393177 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:45.393183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:45.393269 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:45.393391 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:45.394753 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.396765 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.398023 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:45.398469 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:45.398580 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:45.401915 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:45.401920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:45.401923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:45.401960 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:45.405057 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.405811 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.405815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.405817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.405963 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:45.406345 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:45.406349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:45.406351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:45.406727 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:45.406926 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:45.408611 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0812 01:14:45.408866 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260812 01:14:45.414649 12511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318: Bootstrap starting.
I20260812 01:14:45.418007 12511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:45.419304 12511 log.cc:824] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:45.424701 12511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318: No bootstrap required, opened a new log
I20260812 01:14:45.432627 12511 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } }
I20260812 01:14:45.433034 12511 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:45.433171 12511 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fbcf6ffca9433a986a4388e742a318, State: Initialized, Role: FOLLOWER
I20260812 01:14:45.433856 12511 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } }
I20260812 01:14:45.436471 12511 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:14:45.441231 12724 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } } }
I20260812 01:14:45.441757 12724 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 01:14:45.444021 12559 tablet.cc:2369] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:14:45.479715 12398 master_runner.cc:263] No leader master found from master b6fbcf6ffca9433a986a4388e742a318
I20260812 01:14:45.479892 12398 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 01:14:45.481321 12746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:14:45.481644 12746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 01:14:45.649993 12431 tablet.cc:2369] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:14:45.652710 12724 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:14:45.653038 12724 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } }
I20260812 01:14:45.654762 12724 leader_election.cc:288] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0414ba195320412597531b5514359017 (127.20.151.62:43393), 2b8ffa90162d4153ae83666c44cb7db8 (127.20.151.60:45157)
I20260812 01:14:45.656152 12600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6fbcf6ffca9433a986a4388e742a318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b8ffa90162d4153ae83666c44cb7db8" is_pre_election: true
I20260812 01:14:45.656878 12600 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6fbcf6ffca9433a986a4388e742a318 in term 0.
I20260812 01:14:45.657717 12426 leader_election.cc:302] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b8ffa90162d4153ae83666c44cb7db8, b6fbcf6ffca9433a986a4388e742a318; no voters:
I20260812 01:14:45.658262 12724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:14:45.658407 12724 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:14:45.658512 12724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:45.662039 12724 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } }
I20260812 01:14:45.664161 12344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6fbcf6ffca9433a986a4388e742a318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0414ba195320412597531b5514359017" is_pre_election: true
I20260812 01:14:45.664876 12344 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6fbcf6ffca9433a986a4388e742a318 in term 0.
I20260812 01:14:45.672458 12344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6fbcf6ffca9433a986a4388e742a318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0414ba195320412597531b5514359017"
I20260812 01:14:45.672789 12344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:45.674371 12600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6fbcf6ffca9433a986a4388e742a318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b8ffa90162d4153ae83666c44cb7db8"
I20260812 01:14:45.674643 12600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:45.678053 12600 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6fbcf6ffca9433a986a4388e742a318 in term 1.
I20260812 01:14:45.678812 12426 leader_election.cc:302] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b8ffa90162d4153ae83666c44cb7db8, b6fbcf6ffca9433a986a4388e742a318; no voters:
I20260812 01:14:45.681865 12344 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6fbcf6ffca9433a986a4388e742a318 in term 1.
I20260812 01:14:45.686951 12724 leader_election.cc:288] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [CANDIDATE]: Term 1 election: Requested vote from peers 0414ba195320412597531b5514359017 (127.20.151.62:43393), 2b8ffa90162d4153ae83666c44cb7db8 (127.20.151.60:45157)
I20260812 01:14:45.687675 12724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:14:45.688127 12724 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [term 1 LEADER]: Becoming Leader. State: Replica: b6fbcf6ffca9433a986a4388e742a318, State: Running, Role: LEADER
I20260812 01:14:45.689023 12724 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } }
I20260812 01:14:45.709520 12750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fbcf6ffca9433a986a4388e742a318. Latest consensus state: current_term: 1 leader_uuid: "b6fbcf6ffca9433a986a4388e742a318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } } }
I20260812 01:14:45.710170 12750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:45.723918 12753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:14:45.726595 12753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:14:45.746834 12600 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [term 1 FOLLOWER]: Refusing update from remote peer b6fbcf6ffca9433a986a4388e742a318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:14:45.748792 12750 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:14:45.756757 12344 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [term 1 FOLLOWER]: Refusing update from remote peer b6fbcf6ffca9433a986a4388e742a318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:14:45.757862 12750 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:14:45.796754 12677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fbcf6ffca9433a986a4388e742a318. Latest consensus state: current_term: 1 leader_uuid: "b6fbcf6ffca9433a986a4388e742a318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } } }
I20260812 01:14:45.797163 12677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:14:45.802172 12754 mvcc.cc:205] Tried to move back new op lower bound from 7317492882404995072 to 7317492882212786176. Current Snapshot: MvccSnapshot[applied={T|T < 7317492882404995072}]
I20260812 01:14:45.805267 12756 mvcc.cc:205] Tried to move back new op lower bound from 7317492882404995072 to 7317492882212786176. Current Snapshot: MvccSnapshot[applied={T|T < 7317492882404995072}]
I20260812 01:14:45.813436 12750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6fbcf6ffca9433a986a4388e742a318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } } }
I20260812 01:14:45.813764 12750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:45.814345 12753 catalog_manager.cc:1347] Generated new cluster ID: 03452be413e84ef0a9d1a00317cb2874
I20260812 01:14:45.814462 12753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:14:45.814581 12677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6fbcf6ffca9433a986a4388e742a318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } } }
I20260812 01:14:45.814878 12677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:14:45.821800 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fbcf6ffca9433a986a4388e742a318. Latest consensus state: current_term: 1 leader_uuid: "b6fbcf6ffca9433a986a4388e742a318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } } }
I20260812 01:14:45.822137 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:14:45.821959 12724 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6fbcf6ffca9433a986a4388e742a318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } } }
I20260812 01:14:45.822304 12724 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318 [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:45.824642 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6fbcf6ffca9433a986a4388e742a318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0414ba195320412597531b5514359017" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43393 } } peers { permanent_uuid: "b6fbcf6ffca9433a986a4388e742a318" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 44889 } } peers { permanent_uuid: "2b8ffa90162d4153ae83666c44cb7db8" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 45157 } } }
I20260812 01:14:45.824903 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:14:45.825094 12755 mvcc.cc:205] Tried to move back new op lower bound from 7317492882404995072 to 7317492882212786176. Current Snapshot: MvccSnapshot[applied={T|T < 7317492882404995072}]
I20260812 01:14:45.866681 12753 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:14:45.868680 12753 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:14:45.894236 12753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6fbcf6ffca9433a986a4388e742a318: Generated new TSK 0
I20260812 01:14:45.895382 12753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:14:46.357636 12719 catalog_manager.cc:1259] Loaded cluster ID: 03452be413e84ef0a9d1a00317cb2874
I20260812 01:14:46.357764 12719 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017: loading cluster ID for follower catalog manager: success
I20260812 01:14:46.360438 12719 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017: acquiring CA information for follower catalog manager: success
I20260812 01:14:46.362385 12719 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0414ba195320412597531b5514359017: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:14:46.381140 12714 catalog_manager.cc:1259] Loaded cluster ID: 03452be413e84ef0a9d1a00317cb2874
I20260812 01:14:46.381285 12714 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8: loading cluster ID for follower catalog manager: success
I20260812 01:14:46.383965 12714 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8: acquiring CA information for follower catalog manager: success
I20260812 01:14:46.385978 12714 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2b8ffa90162d4153ae83666c44cb7db8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:14:46.482286 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 12276
I20260812 01:14:46.483688 12379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:46.597057 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 12276
I20260812 01:14:46.626536 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 12398
I20260812 01:14:46.627799 12502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:46.739573 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:46.738697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:43393 (local address 127.0.0.1:44394)
0812 01:14:46.739225 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:46.739376 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":256,"threads_started":1}
I20260812 01:14:46.739563 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 12398
I20260812 01:14:46.769647 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 12527
I20260812 01:14:46.771015 12640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:46.882750 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 12527
2026-08-12T01:14:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4838 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:14:47 dist-test-slave-21b9 krb5kdc[12784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:14:47 dist-test-slave-21b9 krb5kdc[12784](info): set up 2 sockets
Aug 12 01:14:47 dist-test-slave-21b9 krb5kdc[12784](info): commencing operation
krb5kdc: starting...
W20260812 01:14:48.172544 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.003s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:14:48 dist-test-slave-21b9 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:14:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:14:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:14:48.502911 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43737
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:45371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:43737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:14:48.858580 12804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:48.858884 12804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:48.859016 12804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:48.866369 12804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:14:48.866504 12804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:48.866585 12804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:48.866662 12804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:14:48.866721 12804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:14:48.877173 12804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:45371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:43737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:48.878649 12804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:48.880039 12804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:48.891996 12819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:48.892323 12813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:48.892355 12816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:48.893479 12804 server_base.cc:1034] running on GCE node
I20260812 01:14:48.894330 12804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:48.895931 12804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:48.897163 12804 hybrid_clock.cc:648] HybridClock initialized: now 1786497288897113 us; error 35 us; skew 500 ppm
I20260812 01:14:48.897555 12804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:48 dist-test-slave-21b9 krb5kdc[12784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497288, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:48.900743 12804 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:14:48.902302 12804 webserver.cc:460] Webserver started at http://127.20.151.62:41425/ using document root <none> and password file <none>
I20260812 01:14:48.902980 12804 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:48.903093 12804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:48.903445 12804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:14:48.906514 12804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "fb75a66b1bd543da9d2ebf4800266159"
format_stamp: "Formatted at 2026-08-12 01:14:48 on dist-test-slave-21b9"
server_key: "b4e833158f150582cd0aeccc531b33a2"
server_key_iv: "a2cf1d8e0cdd7b7f7eeaab411ee0d5f8"
server_key_version: "encryptionkey@0"
I20260812 01:14:48.907258 12804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "fb75a66b1bd543da9d2ebf4800266159"
format_stamp: "Formatted at 2026-08-12 01:14:48 on dist-test-slave-21b9"
server_key: "b4e833158f150582cd0aeccc531b33a2"
server_key_iv: "a2cf1d8e0cdd7b7f7eeaab411ee0d5f8"
server_key_version: "encryptionkey@0"
I20260812 01:14:48.912439 12804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 01:14:48.916067 12831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:48.917363 12804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:14:48.917536 12804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "fb75a66b1bd543da9d2ebf4800266159"
format_stamp: "Formatted at 2026-08-12 01:14:48 on dist-test-slave-21b9"
server_key: "b4e833158f150582cd0aeccc531b33a2"
server_key_iv: "a2cf1d8e0cdd7b7f7eeaab411ee0d5f8"
server_key_version: "encryptionkey@0"
I20260812 01:14:48.917758 12804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:48.939694 12804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:48.943907 12804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:48.944197 12804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:48.962563 12804 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:43737
I20260812 01:14:48.962601 12904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:43737 every 8 connection(s)
I20260812 01:14:48.964015 12804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:14:48.965981 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 12804
I20260812 01:14:48.966215 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:14:48.967065 21084 external_mini_cluster.cc:1428] Setting key 9ec2193fa53f2fa8e720c6e679311988
I20260812 01:14:48.968672 12905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 01:14:48 dist-test-slave-21b9 krb5kdc[12784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:48.982772 12905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159: Bootstrap starting.
I20260812 01:14:48.986223 12905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:48.987612 12905 log.cc:824] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:48.991083 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:48.968997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.20.151.62:43737)
0812 01:14:48.969784 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:48.969798 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:48.969839 (+ 41us) server_negotiation.cc:407] Connection header received
0812 01:14:48.971529 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:48.971551 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:48.971925 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:48.972264 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:48.974155 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:48.975581 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:48.977402 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:48.978039 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:48.978176 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:48.983290 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:48.983301 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:48.983321 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:48.983365 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:48.986414 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:48.987405 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:48.987412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:48.987419 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:48.987616 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:48.988056 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:48.988059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:48.988061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:48.988900 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:48.989203 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:48.989646 (+ 443us) server_negotiation.cc:299] Negotiation successful
0812 01:14:48.990007 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":373,"threads_started":1}
I20260812 01:14:48.991108 12905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159: No bootstrap required, opened a new log
I20260812 01:14:48.995049 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:14:48.999296 12905 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75a66b1bd543da9d2ebf4800266159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43737 } }
I20260812 01:14:48.999734 12905 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:48.999835 12905 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb75a66b1bd543da9d2ebf4800266159, State: Initialized, Role: FOLLOWER
I20260812 01:14:49.000382 12905 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75a66b1bd543da9d2ebf4800266159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43737 } }
I20260812 01:14:49.000617 12905 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:14:49.000749 12905 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:14:49.000910 12905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:49.003175 12905 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75a66b1bd543da9d2ebf4800266159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43737 } }
I20260812 01:14:49.003770 12905 leader_election.cc:302] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb75a66b1bd543da9d2ebf4800266159; no voters:
I20260812 01:14:49.004390 12905 leader_election.cc:288] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:14:49.004527 12916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:14:49.005183 12916 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [term 1 LEADER]: Becoming Leader. State: Replica: fb75a66b1bd543da9d2ebf4800266159, State: Running, Role: LEADER
I20260812 01:14:49.005755 12916 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75a66b1bd543da9d2ebf4800266159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43737 } }
I20260812 01:14:49.006757 12905 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:14:49.008719 12918 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb75a66b1bd543da9d2ebf4800266159. Latest consensus state: current_term: 1 leader_uuid: "fb75a66b1bd543da9d2ebf4800266159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75a66b1bd543da9d2ebf4800266159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43737 } } }
I20260812 01:14:49.009210 12918 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:49.008746 12917 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb75a66b1bd543da9d2ebf4800266159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75a66b1bd543da9d2ebf4800266159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43737 } } }
I20260812 01:14:49.009785 12917 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159 [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:49.015451 12925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:14:49.019891 12925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:14:49.028700 12925 catalog_manager.cc:1347] Generated new cluster ID: 920a7e36e7734299bb072cab50457e89
I20260812 01:14:49.028833 12925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:14:49.037806 12925 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:14:49.039203 12925 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:14:49.048316 12925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb75a66b1bd543da9d2ebf4800266159: Generated new TSK 0
I20260812 01:14:49.049403 12925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:14:49.107362 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 12804
I20260812 01:14:49.108716 12900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:49.207305 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 12804
2026-08-12T01:14:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2320 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:14:49 dist-test-slave-21b9 krb5kdc[12952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:14:49 dist-test-slave-21b9 krb5kdc[12952](info): set up 2 sockets
Aug 12 01:14:49 dist-test-slave-21b9 krb5kdc[12952](info): commencing operation
krb5kdc: starting...
W20260812 01:14:50.488088 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:14:50 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:14:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:14:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:14:50.819945 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46709
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:46709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:14:51.169935 12975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:51.170256 12975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:51.170393 12975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:51.177810 12975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:14:51.177935 12975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:51.177999 12975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:51.178083 12975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:14:51.178148 12975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:14:51.188791 12975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:46709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:51.190231 12975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:51.191596 12975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:51.203385 12984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:51.203435 12983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:51.203917 12991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:51.204427 12975 server_base.cc:1034] running on GCE node
I20260812 01:14:51.205215 12975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:51.206682 12975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:51.207894 12975 hybrid_clock.cc:648] HybridClock initialized: now 1786497291207839 us; error 44 us; skew 500 ppm
I20260812 01:14:51.208258 12975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:51 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497291, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:51.211440 12975 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:14:51.213038 12975 webserver.cc:460] Webserver started at http://127.20.151.62:42617/ using document root <none> and password file <none>
I20260812 01:14:51.213752 12975 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:51.213878 12975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:51.214190 12975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:14:51.217424 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "2bd08a94670d4f949710c2db4217a847"
format_stamp: "Formatted at 2026-08-12 01:14:51 on dist-test-slave-21b9"
server_key: "dc551acb12969772c7ad90593cb9234f"
server_key_iv: "2b23465d4c26854470b36a0799a7376a"
server_key_version: "encryptionkey@0"
I20260812 01:14:51.218449 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "2bd08a94670d4f949710c2db4217a847"
format_stamp: "Formatted at 2026-08-12 01:14:51 on dist-test-slave-21b9"
server_key: "dc551acb12969772c7ad90593cb9234f"
server_key_iv: "2b23465d4c26854470b36a0799a7376a"
server_key_version: "encryptionkey@0"
I20260812 01:14:51.224284 12975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 01:14:51.228073 13002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:51.229533 12975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 01:14:51.229712 12975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "2bd08a94670d4f949710c2db4217a847"
format_stamp: "Formatted at 2026-08-12 01:14:51 on dist-test-slave-21b9"
server_key: "dc551acb12969772c7ad90593cb9234f"
server_key_iv: "2b23465d4c26854470b36a0799a7376a"
server_key_version: "encryptionkey@0"
I20260812 01:14:51.229905 12975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:51.249733 12975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:51.255352 12975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:51.255703 12975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:51.276034 12975 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:46709
I20260812 01:14:51.276100 13083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:46709 every 8 connection(s)
I20260812 01:14:51.277495 12975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:14:51.282238 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:51.283161 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 12975
I20260812 01:14:51.283401 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:14:51.284078 21084 external_mini_cluster.cc:1428] Setting key f67f30e138bcbd58ed87ba7316930965
Aug 12 01:14:51 dist-test-slave-21b9 krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:51.298874 13085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Bootstrap starting.
I20260812 01:14:51.303311 13085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:51.304729 13085 log.cc:824] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:51.307451 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:51.286220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.20.151.62:46709)
0812 01:14:51.286993 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:51.287006 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:51.287048 (+ 42us) server_negotiation.cc:407] Connection header received
0812 01:14:51.289535 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:51.289561 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:51.290020 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:51.290442 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:51.292048 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:51.293354 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:51.294704 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:51.295498 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:51.295682 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:51.300081 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:51.300091 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:51.300107 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:51.300147 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:51.303162 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:51.303949 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:51.303956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:51.303963 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:51.304146 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:51.304536 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:51.304540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:51.304542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:51.305161 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:51.305489 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:51.305948 (+ 459us) server_negotiation.cc:299] Negotiation successful
0812 01:14:51.306419 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":346,"threads_started":1}
I20260812 01:14:51.308281 13085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: No bootstrap required, opened a new log
I20260812 01:14:51.319573 13085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:51.320204 13085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:51.320381 13085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd08a94670d4f949710c2db4217a847, State: Initialized, Role: FOLLOWER
I20260812 01:14:51.321115 13085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:51.321398 13085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:14:51.321532 13085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:14:51.321725 13085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:51.325066 13085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:51.325730 13085 leader_election.cc:302] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bd08a94670d4f949710c2db4217a847; no voters:
I20260812 01:14:51.326457 13085 leader_election.cc:288] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:14:51.326587 13091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:14:51.327854 13091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 1 LEADER]: Becoming Leader. State: Replica: 2bd08a94670d4f949710c2db4217a847, State: Running, Role: LEADER
I20260812 01:14:51.328552 13091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:51.328835 13085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:14:51.331493 13094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bd08a94670d4f949710c2db4217a847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } } }
I20260812 01:14:51.331861 13094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:51.333191 13095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bd08a94670d4f949710c2db4217a847. Latest consensus state: current_term: 1 leader_uuid: "2bd08a94670d4f949710c2db4217a847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } } }
I20260812 01:14:51.333626 13095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:51.335569 13100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:14:51.341490 13100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:14:51.351970 13100 catalog_manager.cc:1347] Generated new cluster ID: 5a08f33c02ee45dc85c3431cec4e893a
I20260812 01:14:51.352110 13100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:14:51.370273 13100 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:14:51.371692 13100 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:14:51.377310 13100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Generated new TSK 0
I20260812 01:14:51.378240 13100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:14:51.469048 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46709
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:14:51.827562 13119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:51.827864 13119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:51.827991 13119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:51.835834 13119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:51.835951 13119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:51.836100 13119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:14:51.845980 13119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:51.847353 13119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:51.848683 13119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:51.862005 13128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:51.861928 13130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:51.861967 13132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:51.862668 13119 server_base.cc:1034] running on GCE node
I20260812 01:14:51.863299 13119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:51.864281 13119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:51.865520 13119 hybrid_clock.cc:648] HybridClock initialized: now 1786497291865482 us; error 29 us; skew 500 ppm
I20260812 01:14:51.865830 13119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:51 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497291, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:51.869033 13119 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:14:51.870539 13119 webserver.cc:460] Webserver started at http://127.20.151.1:37745/ using document root <none> and password file <none>
I20260812 01:14:51.871219 13119 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:51.871371 13119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:51.871681 13119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:14:51.874634 13119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "cd60e7746a0447278e642cd1ef1b6d6e"
format_stamp: "Formatted at 2026-08-12 01:14:51 on dist-test-slave-21b9"
server_key: "22967cbb0a5e2a086ccf99c80bf7c8cf"
server_key_iv: "9db4090da819153d639469bef9f7d9d8"
server_key_version: "encryptionkey@0"
I20260812 01:14:51.875455 13119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "cd60e7746a0447278e642cd1ef1b6d6e"
format_stamp: "Formatted at 2026-08-12 01:14:51 on dist-test-slave-21b9"
server_key: "22967cbb0a5e2a086ccf99c80bf7c8cf"
server_key_iv: "9db4090da819153d639469bef9f7d9d8"
server_key_version: "encryptionkey@0"
I20260812 01:14:51.880810 13119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 01:14:51.884287 13140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:51.885605 13119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:14:51.885766 13119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "cd60e7746a0447278e642cd1ef1b6d6e"
format_stamp: "Formatted at 2026-08-12 01:14:51 on dist-test-slave-21b9"
server_key: "22967cbb0a5e2a086ccf99c80bf7c8cf"
server_key_iv: "9db4090da819153d639469bef9f7d9d8"
server_key_version: "encryptionkey@0"
I20260812 01:14:51.885969 13119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:51.913409 13119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:51.917799 13119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:51.918099 13119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:51.919145 13119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:51.920686 13119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:51.920771 13119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:51.920886 13119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:51.920939 13119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:51.953330 13119 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:45173
I20260812 01:14:51.953410 13333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:45173 every 8 connection(s)
I20260812 01:14:51.954648 13119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:14:51.962055 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 13119
I20260812 01:14:51.962247 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:14:51.962935 21084 external_mini_cluster.cc:1428] Setting key 08bc56912074002246e5b3e221dde2e5
Aug 12 01:14:51 dist-test-slave-21b9 krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497291, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:51.974627 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:51.957272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:50543 (local address 127.20.151.62:46709)
0812 01:14:51.957706 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:51.957711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:51.958691 (+ 980us) server_negotiation.cc:407] Connection header received
0812 01:14:51.960575 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:51.960586 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:51.960676 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:51.960777 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:51.962478 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:51.963250 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:51.965068 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:51.965466 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:51.965591 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:51.969019 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:51.969025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:51.969028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:51.969070 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:51.971561 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:51.972286 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:51.972291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:51.972293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:51.972417 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:51.972839 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:51.972844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:51.972846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:51.973130 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:51.973267 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:51.973989 (+ 722us) server_negotiation.cc:299] Negotiation successful
0812 01:14:51.974235 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
I20260812 01:14:51.975663 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:51.957833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:50543)
0812 01:14:51.958524 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:51.958566 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:51.960298 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:51.961046 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:51.961055 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:51.961623 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:51.962283 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:51.962312 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:51.963438 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:51.963443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:51.964922 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:51.964935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:51.965699 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:51.965706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:51.965930 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:51.966595 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:51.966618 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:51.968796 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:51.971702 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:51.971711 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:51.971727 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:51.972173 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:51.972526 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:51.972531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:51.972535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:51.972731 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:51.973263 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:51.973272 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:51.973611 (+ 339us) client_negotiation.cc:769] Sending connection context
0812 01:14:51.973866 (+ 255us) client_negotiation.cc:240] Negotiation successful
0812 01:14:51.974436 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":271,"threads_started":1}
I20260812 01:14:51.977332 13334 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46709
I20260812 01:14:51.977636 13334 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:51.978369 13334 heartbeater.cc:511] Master 127.20.151.62:46709 requested a full tablet report, sending...
I20260812 01:14:51.980600 13032 ts_manager.cc:194] Registered new tserver with Master: cd60e7746a0447278e642cd1ef1b6d6e (127.20.151.1:45173)
I20260812 01:14:51.982049 13032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:50543
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:14:52.035095 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46709
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:14:52.386747 13342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:52.387074 13342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:52.387217 13342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:52.394858 13342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:52.395020 13342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:52.395181 13342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:14:52.405617 13342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:52.407012 13342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:52.408443 13342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:52.422077 13349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:52.422030 13351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:52.422149 13355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:52.423142 13342 server_base.cc:1034] running on GCE node
I20260812 01:14:52.423861 13342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:52.424954 13342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:52.426265 13342 hybrid_clock.cc:648] HybridClock initialized: now 1786497292426162 us; error 83 us; skew 500 ppm
I20260812 01:14:52.426680 13342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:52 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497292, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:52.430259 13342 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:14:52.431926 13342 webserver.cc:460] Webserver started at http://127.20.151.2:46473/ using document root <none> and password file <none>
I20260812 01:14:52.432654 13342 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:52.432783 13342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:52.433130 13342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:14:52.436358 13342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "8ed72a3b1d5d45988a1bb05914e5595c"
format_stamp: "Formatted at 2026-08-12 01:14:52 on dist-test-slave-21b9"
server_key: "aa9a83d3d0e15a35c8611a7507c3b320"
server_key_iv: "9bd428cb57f7d565ed84b4cc661c0da1"
server_key_version: "encryptionkey@0"
I20260812 01:14:52.437218 13342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "8ed72a3b1d5d45988a1bb05914e5595c"
format_stamp: "Formatted at 2026-08-12 01:14:52 on dist-test-slave-21b9"
server_key: "aa9a83d3d0e15a35c8611a7507c3b320"
server_key_iv: "9bd428cb57f7d565ed84b4cc661c0da1"
server_key_version: "encryptionkey@0"
I20260812 01:14:52.442510 13342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260812 01:14:52.446015 13367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:52.447618 13342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:14:52.447794 13342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "8ed72a3b1d5d45988a1bb05914e5595c"
format_stamp: "Formatted at 2026-08-12 01:14:52 on dist-test-slave-21b9"
server_key: "aa9a83d3d0e15a35c8611a7507c3b320"
server_key_iv: "9bd428cb57f7d565ed84b4cc661c0da1"
server_key_version: "encryptionkey@0"
I20260812 01:14:52.448009 13342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:52.480473 13342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:52.484962 13342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:52.485261 13342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:52.486320 13342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:52.487787 13342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:52.487872 13342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:52.487998 13342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:52.488059 13342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:52.520221 13342 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:36217
I20260812 01:14:52.520280 13567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:36217 every 8 connection(s)
I20260812 01:14:52.521550 13342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:14:52.528757 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 13342
I20260812 01:14:52.528939 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:14:52.529516 21084 external_mini_cluster.cc:1428] Setting key 80b0a9f9facb701fe24b305f2de9990a
Aug 12 01:14:52 dist-test-slave-21b9 krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497292, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:52.543689 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:52.524124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:34915 (local address 127.20.151.62:46709)
0812 01:14:52.524604 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:52.524611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:52.525625 (+ 1014us) server_negotiation.cc:407] Connection header received
0812 01:14:52.527522 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:52.527527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:52.527614 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:52.527699 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:52.529458 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:52.530477 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:52.532626 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:52.533174 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:52.533302 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:52.537077 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:52.537084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:52.537088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:52.537128 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:52.540367 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:52.541188 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:52.541193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:52.541196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:52.541358 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:52.541801 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:52.541805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:52.541807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:52.542163 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:52.542310 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:52.543063 (+ 753us) server_negotiation.cc:299] Negotiation successful
0812 01:14:52.543329 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":227,"threads_started":1}
I20260812 01:14:52.544587 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:52.524676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:34915)
0812 01:14:52.525460 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:52.525500 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:52.527193 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:52.527936 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:52.527945 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:52.528514 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:52.529239 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:52.529260 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:52.530700 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:52.530705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:52.532457 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:52.532471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:52.533387 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:52.533393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:52.533652 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:52.534367 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:52.534392 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:52.536817 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:52.540526 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:52.540535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:52.540551 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:52.541043 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:52.541458 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:52.541461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:52.541464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:52.541690 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:52.542288 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:52.542295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:52.542664 (+ 369us) client_negotiation.cc:769] Sending connection context
0812 01:14:52.542921 (+ 257us) client_negotiation.cc:240] Negotiation successful
0812 01:14:52.543395 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":302,"threads_started":1}
I20260812 01:14:52.546058 13571 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46709
I20260812 01:14:52.546401 13571 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:52.547186 13571 heartbeater.cc:511] Master 127.20.151.62:46709 requested a full tablet report, sending...
I20260812 01:14:52.548771 13032 ts_manager.cc:194] Registered new tserver with Master: 8ed72a3b1d5d45988a1bb05914e5595c (127.20.151.2:36217)
I20260812 01:14:52.549611 13032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:34915
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:14:52.600986 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46709
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:14:52.951574 13577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:52.951895 13577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:52.952034 13577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:52.959843 13577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:52.959972 13577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:52.960130 13577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:14:52.970027 13577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:52.971402 13577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:52.972797 13577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 01:14:52.985137 13334 heartbeater.cc:503] Master 127.20.151.62:46709 was elected leader, sending a full tablet report...
W20260812 01:14:52.985653 13583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:52.985713 13582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:52.985755 13587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:52.987530 13577 server_base.cc:1034] running on GCE node
I20260812 01:14:52.988169 13577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:52.989182 13577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:52.990469 13577 hybrid_clock.cc:648] HybridClock initialized: now 1786497292990363 us; error 46 us; skew 500 ppm
I20260812 01:14:52.990824 13577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:52 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497292, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:52.994074 13577 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:14:52.995700 13577 webserver.cc:460] Webserver started at http://127.20.151.3:34751/ using document root <none> and password file <none>
I20260812 01:14:52.996477 13577 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:52.996595 13577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:52.996861 13577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:14:53.001627 13577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "1fc0fa08bf1d493887cde9461b7c8a30"
format_stamp: "Formatted at 2026-08-12 01:14:52 on dist-test-slave-21b9"
server_key: "f31276f95b0774db6843b3ea7dda32ef"
server_key_iv: "6c9c51389638d76bed03939fbf9d7568"
server_key_version: "encryptionkey@0"
I20260812 01:14:53.003101 13577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "1fc0fa08bf1d493887cde9461b7c8a30"
format_stamp: "Formatted at 2026-08-12 01:14:52 on dist-test-slave-21b9"
server_key: "f31276f95b0774db6843b3ea7dda32ef"
server_key_iv: "6c9c51389638d76bed03939fbf9d7568"
server_key_version: "encryptionkey@0"
I20260812 01:14:53.010872 13577 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 01:14:53.015973 13595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:53.017690 13577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 01:14:53.017891 13577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "1fc0fa08bf1d493887cde9461b7c8a30"
format_stamp: "Formatted at 2026-08-12 01:14:52 on dist-test-slave-21b9"
server_key: "f31276f95b0774db6843b3ea7dda32ef"
server_key_iv: "6c9c51389638d76bed03939fbf9d7568"
server_key_version: "encryptionkey@0"
I20260812 01:14:53.018208 13577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:53.046469 13577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:53.052244 13577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:53.052587 13577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:53.053860 13577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:53.055708 13577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:53.055807 13577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:53.055940 13577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:53.056003 13577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:53.089510 13577 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:33415
I20260812 01:14:53.089478 13780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:33415 every 8 connection(s)
I20260812 01:14:53.091394 13577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:14:53.095245 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 13577
I20260812 01:14:53.095499 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:14:53.096113 21084 external_mini_cluster.cc:1428] Setting key d9385cd3712d5ef1426999c057f018c5
Aug 12 01:14:53 dist-test-slave-21b9 krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497292, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:53.113642 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.094381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:50563 (local address 127.20.151.62:46709)
0812 01:14:53.094936 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:53.094943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:53.095906 (+ 963us) server_negotiation.cc:407] Connection header received
0812 01:14:53.098188 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:53.098194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:53.098281 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:53.098375 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:53.100324 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.101158 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.102741 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.103231 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.103374 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.107210 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:53.107215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:53.107219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:53.107254 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:53.109950 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:53.110807 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:53.110812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:53.110815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:53.111006 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:53.111622 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:53.111638 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:53.111642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:53.112008 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:53.112169 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:53.113141 (+ 972us) server_negotiation.cc:299] Negotiation successful
0812 01:14:53.113322 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":240,"threads_started":1}
I20260812 01:14:53.115127 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.094899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:50563)
0812 01:14:53.095740 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:53.095794 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:53.097906 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:53.098651 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:53.098665 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:53.099483 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:53.100158 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.100177 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.101312 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.101316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.102596 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.102609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.103476 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.103484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.103720 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.104367 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:53.104392 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:53.106924 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:53.110100 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:53.110110 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:53.110129 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:53.110693 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:53.111193 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:53.111197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:53.111200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:53.111464 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:53.112166 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:53.112176 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:53.112685 (+ 509us) client_negotiation.cc:769] Sending connection context
0812 01:14:53.113049 (+ 364us) client_negotiation.cc:240] Negotiation successful
0812 01:14:53.113653 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":340,"threads_started":1}
I20260812 01:14:53.116555 13781 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46709
I20260812 01:14:53.116906 13781 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:53.117844 13781 heartbeater.cc:511] Master 127.20.151.62:46709 requested a full tablet report, sending...
I20260812 01:14:53.119555 13032 ts_manager.cc:194] Registered new tserver with Master: 1fc0fa08bf1d493887cde9461b7c8a30 (127.20.151.3:33415)
I20260812 01:14:53.120368 13032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:50563
I20260812 01:14:53.124993 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:14:53.143132 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.129699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.20.151.62:46709)
0812 01:14:53.129890 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:53.129897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:53.130468 (+ 571us) server_negotiation.cc:407] Connection header received
0812 01:14:53.130520 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:53.130523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:53.130592 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:53.130670 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:53.132159 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.132908 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.134374 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.134843 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.134944 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.137505 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:53.137510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:53.137514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:53.137570 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:53.140592 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:53.141234 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:53.141240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:53.141244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:53.141376 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:53.141741 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:53.141746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:53.141750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:53.142074 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:53.142299 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:53.142627 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 01:14:53.142840 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 01:14:53.147626 13032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:14:53.158464 13032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:14:53.190711 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.173918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:45173 (local address 127.0.0.1:36866)
0812 01:14:53.177128 (+ 3210us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:53.177146 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:53.177264 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:53.178384 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:53.178393 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:53.178439 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:53.180116 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.180136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.182294 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.182300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.186118 (+ 3818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.186139 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.189652 (+ 3513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.189659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.189871 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.189925 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 01:14:53.190060 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 01:14:53.190274 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3103,"thread_start_us":2991,"threads_started":1}
I20260812 01:14:53.191802 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.174506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.20.151.1:45173)
0812 01:14:53.176736 (+ 2230us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:53.176746 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:53.177390 (+ 644us) server_negotiation.cc:407] Connection header received
0812 01:14:53.177539 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:53.177549 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:53.177836 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:53.178050 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:53.180352 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.182102 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.187415 (+ 5313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.189099 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.189270 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.189523 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:53.191250 (+ 1727us) server_negotiation.cc:299] Negotiation successful
0812 01:14:53.191477 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2059,"thread_start_us":1909,"threads_started":1}
I20260812 01:14:53.195880 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.172526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:36217 (local address 127.0.0.1:42140)
0812 01:14:53.174871 (+ 2345us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:53.174889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:53.176147 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:53.187733 (+ 11586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:53.187737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:53.187760 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:53.188196 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.188208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.190501 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.190506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.192972 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.192989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.195053 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.195059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.195233 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.195265 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:14:53.195353 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 01:14:53.195491 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2232,"mutex_wait_us":2,"thread_start_us":1165,"threads_started":1}
I20260812 01:14:53.196436 13228 tablet_service.cc:1467] Processing CreateTablet for tablet a38b40f1413642b1b8bf28bbec21b77a (DEFAULT_TABLE table=test-table [id=9cd02f001457481899640cbda93ac644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:14:53.197126 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.173053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.20.151.2:36217)
0812 01:14:53.187138 (+ 14085us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:53.187146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:53.187173 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:14:53.187264 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:53.187298 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:53.187518 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:53.188387 (+ 869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:53.188648 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.190331 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.193174 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.194787 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.196325 (+ 1538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.196534 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:53.196604 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 01:14:53.196766 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13914,"thread_start_us":13781,"threads_started":1}
I20260812 01:14:53.197765 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.172292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:33415 (local address 127.0.0.1:48516)
0812 01:14:53.174127 (+ 1835us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:53.174172 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:53.174365 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:53.187515 (+ 13150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:53.187521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:53.187548 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:53.188026 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.188043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.191838 (+ 3795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.191844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.193854 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.193869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.197052 (+ 3183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.197059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.197224 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.197258 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 01:14:53.197325 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 01:14:53.197467 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1544,"thread_start_us":3261,"threads_started":1}
I20260812 01:14:53.198101 13228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38b40f1413642b1b8bf28bbec21b77a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:53.198623 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.173055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.20.151.3:33415)
0812 01:14:53.186792 (+ 13737us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:53.186801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:53.186840 (+ 39us) server_negotiation.cc:407] Connection header received
0812 01:14:53.186945 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:53.186953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:53.187241 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:53.189489 (+ 2248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:53.189917 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.191667 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.194131 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.196832 (+ 2701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.197571 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.197800 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:53.198049 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 01:14:53.198286 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13553,"thread_start_us":13394,"threads_started":1}
I20260812 01:14:53.201922 13443 tablet_service.cc:1467] Processing CreateTablet for tablet a38b40f1413642b1b8bf28bbec21b77a (DEFAULT_TABLE table=test-table [id=9cd02f001457481899640cbda93ac644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:14:53.202242 13674 tablet_service.cc:1467] Processing CreateTablet for tablet a38b40f1413642b1b8bf28bbec21b77a (DEFAULT_TABLE table=test-table [id=9cd02f001457481899640cbda93ac644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:14:53.203540 13674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38b40f1413642b1b8bf28bbec21b77a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:53.205242 13443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38b40f1413642b1b8bf28bbec21b77a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:53.220293 13814 tablet_bootstrap.cc:492] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e: Bootstrap starting.
I20260812 01:14:53.226236 13814 tablet_bootstrap.cc:654] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:53.227650 13814 log.cc:824] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:53.228974 13816 tablet_bootstrap.cc:492] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: Bootstrap starting.
I20260812 01:14:53.230270 13815 tablet_bootstrap.cc:492] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: Bootstrap starting.
I20260812 01:14:53.233362 13816 tablet_bootstrap.cc:654] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:53.233891 13815 tablet_bootstrap.cc:654] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:53.234781 13814 tablet_bootstrap.cc:492] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e: No bootstrap required, opened a new log
I20260812 01:14:53.234838 13816 log.cc:824] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:53.235193 13814 ts_tablet_manager.cc:1397] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20260812 01:14:53.235183 13815 log.cc:824] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:53.244436 13816 tablet_bootstrap.cc:492] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: No bootstrap required, opened a new log
I20260812 01:14:53.244762 13816 ts_tablet_manager.cc:1397] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260812 01:14:53.246860 13814 raft_consensus.cc:348] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:53.247539 13814 raft_consensus.cc:374] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:53.247692 13814 raft_consensus.cc:729] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd60e7746a0447278e642cd1ef1b6d6e, State: Initialized, Role: FOLLOWER
I20260812 01:14:53.248302 13814 consensus_queue.cc:260] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:53.250391 13815 tablet_bootstrap.cc:492] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: No bootstrap required, opened a new log
I20260812 01:14:53.250773 13815 ts_tablet_manager.cc:1397] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20260812 01:14:53.256177 13816 raft_consensus.cc:348] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:53.256667 13816 raft_consensus.cc:374] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:53.256788 13816 raft_consensus.cc:729] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed72a3b1d5d45988a1bb05914e5595c, State: Initialized, Role: FOLLOWER
I20260812 01:14:53.257742 13816 consensus_queue.cc:260] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:53.266119 13816 ts_tablet_manager.cc:1428] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: Time spent starting tablet: real 0.021s user 0.021s sys 0.000s
I20260812 01:14:53.265866 13815 raft_consensus.cc:348] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:53.266566 13815 raft_consensus.cc:374] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:53.266574 13571 heartbeater.cc:503] Master 127.20.151.62:46709 was elected leader, sending a full tablet report...
I20260812 01:14:53.266721 13815 raft_consensus.cc:729] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc0fa08bf1d493887cde9461b7c8a30, State: Initialized, Role: FOLLOWER
I20260812 01:14:53.267817 13815 consensus_queue.cc:260] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:53.269521 13814 ts_tablet_manager.cc:1428] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e: Time spent starting tablet: real 0.034s user 0.006s sys 0.027s
W20260812 01:14:53.273689 13572 tablet.cc:2369] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:14:53.279732 13781 heartbeater.cc:503] Master 127.20.151.62:46709 was elected leader, sending a full tablet report...
I20260812 01:14:53.280627 13815 ts_tablet_manager.cc:1428] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: Time spent starting tablet: real 0.030s user 0.011s sys 0.009s
W20260812 01:14:53.343387 13782 tablet.cc:2369] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:14:53.356432 13825 raft_consensus.cc:482] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:14:53.356740 13825 raft_consensus.cc:504] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:53.358459 13825 leader_election.cc:288] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fc0fa08bf1d493887cde9461b7c8a30 (127.20.151.3:33415), 8ed72a3b1d5d45988a1bb05914e5595c (127.20.151.2:36217)
I20260812 01:14:53.367624 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.359007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:45135 (local address 127.20.151.3:33415)
0812 01:14:53.359218 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:53.359224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:53.360064 (+ 840us) server_negotiation.cc:407] Connection header received
0812 01:14:53.360256 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:53.360261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:53.360345 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:53.360445 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:53.361261 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.362426 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.365472 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.366699 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.366873 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.366953 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:53.367219 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 01:14:53.367378 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:14:53.367769 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.358970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:33415 (local address 127.20.151.1:45135)
0812 01:14:53.359955 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:53.359971 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:53.360115 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:53.360486 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:53.360491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:53.360535 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:53.361044 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.361057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.362591 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.362596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.365291 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.365307 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.366907 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.366912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.367038 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.367066 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 01:14:53.367157 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 01:14:53.367338 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":246,"spinlock_wait_cycles":5888,"thread_start_us":523,"threads_started":1}
I20260812 01:14:53.367986 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.359074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:37521 (local address 127.20.151.2:36217)
0812 01:14:53.359251 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:53.359256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:53.360063 (+ 807us) server_negotiation.cc:407] Connection header received
0812 01:14:53.360277 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:53.360282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:53.360354 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:53.360448 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:53.361261 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.362426 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.365477 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.366742 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.366889 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.366954 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:53.367579 (+ 625us) server_negotiation.cc:299] Negotiation successful
0812 01:14:53.367725 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:14:53.368685 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.358970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:36217 (local address 127.20.151.1:37521)
0812 01:14:53.359915 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:53.359937 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:53.360157 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:53.360486 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:53.360491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:53.360526 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:53.361042 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.361057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.362590 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.362596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.365289 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:53.365307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.367152 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.367159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:53.367358 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.367391 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 01:14:53.368177 (+ 786us) client_negotiation.cc:240] Negotiation successful
0812 01:14:53.368321 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":211,"spinlock_wait_cycles":41984,"thread_start_us":462,"threads_started":1}
I20260812 01:14:53.369277 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38b40f1413642b1b8bf28bbec21b77a" candidate_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" is_pre_election: true
I20260812 01:14:53.369676 13482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38b40f1413642b1b8bf28bbec21b77a" candidate_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" is_pre_election: true
I20260812 01:14:53.369830 13705 raft_consensus.cc:2454] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd60e7746a0447278e642cd1ef1b6d6e in term 0.
I20260812 01:14:53.370141 13482 raft_consensus.cc:2454] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd60e7746a0447278e642cd1ef1b6d6e in term 0.
I20260812 01:14:53.370541 13144 leader_election.cc:302] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc0fa08bf1d493887cde9461b7c8a30, cd60e7746a0447278e642cd1ef1b6d6e; no voters:
I20260812 01:14:53.371040 13825 raft_consensus.cc:2781] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:14:53.371210 13825 raft_consensus.cc:482] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:14:53.371342 13825 raft_consensus.cc:3037] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:53.373709 13825 raft_consensus.cc:504] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:53.374303 13825 leader_election.cc:288] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [CANDIDATE]: Term 1 election: Requested vote from peers 1fc0fa08bf1d493887cde9461b7c8a30 (127.20.151.3:33415), 8ed72a3b1d5d45988a1bb05914e5595c (127.20.151.2:36217)
I20260812 01:14:53.374876 13482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38b40f1413642b1b8bf28bbec21b77a" candidate_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed72a3b1d5d45988a1bb05914e5595c"
I20260812 01:14:53.374879 13705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38b40f1413642b1b8bf28bbec21b77a" candidate_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc0fa08bf1d493887cde9461b7c8a30"
I20260812 01:14:53.375069 13705 raft_consensus.cc:3037] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:53.375077 13482 raft_consensus.cc:3037] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:53.377326 13482 raft_consensus.cc:2454] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd60e7746a0447278e642cd1ef1b6d6e in term 1.
I20260812 01:14:53.377351 13705 raft_consensus.cc:2454] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd60e7746a0447278e642cd1ef1b6d6e in term 1.
I20260812 01:14:53.377795 13144 leader_election.cc:302] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc0fa08bf1d493887cde9461b7c8a30, cd60e7746a0447278e642cd1ef1b6d6e; no voters:
I20260812 01:14:53.378151 13825 raft_consensus.cc:2781] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:14:53.378630 13825 raft_consensus.cc:686] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 1 LEADER]: Becoming Leader. State: Replica: cd60e7746a0447278e642cd1ef1b6d6e, State: Running, Role: LEADER
I20260812 01:14:53.379114 13825 consensus_queue.cc:237] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:53.384086 13031 catalog_manager.cc:5495] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e reported cstate change: term changed from 0 to 1, leader changed from <none> to cd60e7746a0447278e642cd1ef1b6d6e (127.20.151.1). New cstate: current_term: 1 leader_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } health_report { overall_health: HEALTHY } } }
I20260812 01:14:53.426520 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.421167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.20.151.1:45173)
0812 01:14:53.421346 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:53.421351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:53.421369 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:14:53.421504 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:53.421508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:53.421612 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:53.421744 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:14:53.422078 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.422798 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.424303 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.424784 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.424883 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.425446 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:14:53.426008 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:14:53.426122 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:53.426223 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 01:14:53.426299 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 01:14:53.435305 13705 raft_consensus.cc:1261] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30 [term 1 FOLLOWER]: Refusing update from remote peer cd60e7746a0447278e642cd1ef1b6d6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:14:53.435305 13482 raft_consensus.cc:1261] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c [term 1 FOLLOWER]: Refusing update from remote peer cd60e7746a0447278e642cd1ef1b6d6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:14:53.436479 13825 consensus_queue.cc:1035] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:14:53.436846 13832 consensus_queue.cc:1035] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:14:53.445912 13836 mvcc.cc:205] Tried to move back new op lower bound from 7317492913899782144 to 7317492913684578304. Current Snapshot: MvccSnapshot[applied={T|T < 7317492913899782144}]
I20260812 01:14:53.447438 13837 mvcc.cc:205] Tried to move back new op lower bound from 7317492913899782144 to 7317492913684578304. Current Snapshot: MvccSnapshot[applied={T|T < 7317492913899782144}]
I20260812 01:14:53.460155 13031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:14:53.460475 13031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:14:53.465472 13031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Sending DeleteTablet for 3 replicas of tablet a38b40f1413642b1b8bf28bbec21b77a
I20260812 01:14:53.466627 13674 tablet_service.cc:1514] Processing DeleteTablet for tablet a38b40f1413642b1b8bf28bbec21b77a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:14:53 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:48516
I20260812 01:14:53.466782 13443 tablet_service.cc:1514] Processing DeleteTablet for tablet a38b40f1413642b1b8bf28bbec21b77a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:14:53 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:42140
I20260812 01:14:53.467981 13442 tablet_service.cc:1514] Processing DeleteTablet for tablet a38b40f1413642b1b8bf28bbec21b77a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:14:53 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:42140
I20260812 01:14:53.468506 13228 tablet_service.cc:1514] Processing DeleteTablet for tablet a38b40f1413642b1b8bf28bbec21b77a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:14:53 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:36866
I20260812 01:14:53.476454 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 13119
I20260812 01:14:53.477768 13857 tablet_replica.cc:330] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e: stopping tablet replica
I20260812 01:14:53.478410 13857 raft_consensus.cc:2229] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:14:53.479362 13857 raft_consensus.cc:2258] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:14:53.480543 13855 tablet_replica.cc:330] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: stopping tablet replica
I20260812 01:14:53.480778 13854 tablet_replica.cc:330] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: stopping tablet replica
I20260812 01:14:53.481142 13855 raft_consensus.cc:2229] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:14:53.481148 13854 raft_consensus.cc:2229] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:14:53.481654 13855 raft_consensus.cc:2258] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:14:53.481665 13854 raft_consensus.cc:2258] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:14:53.483882 13855 ts_tablet_manager.cc:1905] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:14:53.484404 13854 ts_tablet_manager.cc:1905] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:14:53.488852 13857 ts_tablet_manager.cc:1905] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:14:53.491451 13855 ts_tablet_manager.cc:1918] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:14:53.491729 13855 log.cc:1196] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/a38b40f1413642b1b8bf28bbec21b77a
I20260812 01:14:53.492342 13855 ts_tablet_manager.cc:1939] T a38b40f1413642b1b8bf28bbec21b77a P 8ed72a3b1d5d45988a1bb05914e5595c: Deleting consensus metadata
Aug 12 01:14:53 dist-test-slave-21b9 krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:14:53.494525 13003 catalog_manager.cc:4841] TS 8ed72a3b1d5d45988a1bb05914e5595c (127.20.151.2:36217): tablet a38b40f1413642b1b8bf28bbec21b77a (table test-table [id=9cd02f001457481899640cbda93ac644]) successfully deleted
W20260812 01:14:53.495214 13003 catalog_manager.cc:4808] TS 8ed72a3b1d5d45988a1bb05914e5595c (127.20.151.2:36217): delete failed for tablet a38b40f1413642b1b8bf28bbec21b77a because the tablet was not found. No further retry: Not found: Tablet not found: a38b40f1413642b1b8bf28bbec21b77a
I20260812 01:14:53.496412 13857 ts_tablet_manager.cc:1918] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:14:53.496624 13857 log.cc:1196] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/a38b40f1413642b1b8bf28bbec21b77a
I20260812 01:14:53.497242 13857 ts_tablet_manager.cc:1939] T a38b40f1413642b1b8bf28bbec21b77a P cd60e7746a0447278e642cd1ef1b6d6e: Deleting consensus metadata
I20260812 01:14:53.497850 13854 ts_tablet_manager.cc:1918] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:14:53.498095 13854 log.cc:1196] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/a38b40f1413642b1b8bf28bbec21b77a
I20260812 01:14:53.498716 13854 ts_tablet_manager.cc:1939] T a38b40f1413642b1b8bf28bbec21b77a P 1fc0fa08bf1d493887cde9461b7c8a30: Deleting consensus metadata
I20260812 01:14:53.499199 13010 catalog_manager.cc:4841] TS cd60e7746a0447278e642cd1ef1b6d6e (127.20.151.1:45173): tablet a38b40f1413642b1b8bf28bbec21b77a (table test-table [id=9cd02f001457481899640cbda93ac644]) successfully deleted
I20260812 01:14:53.500422 13008 catalog_manager.cc:4841] TS 1fc0fa08bf1d493887cde9461b7c8a30 (127.20.151.3:33415): tablet a38b40f1413642b1b8bf28bbec21b77a (table test-table [id=9cd02f001457481899640cbda93ac644]) successfully deleted
I20260812 01:14:53.504372 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.478008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.20.151.1:45173)
0812 01:14:53.478217 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:53.478224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:53.482557 (+ 4333us) server_negotiation.cc:407] Connection header received
0812 01:14:53.482784 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:53.482790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:53.482874 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:53.482985 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:53.485062 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.486304 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.488486 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.489241 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.489362 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.496451 (+ 7089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:53.496467 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:53.496486 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:53.496536 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:53.501108 (+ 4572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:53.501995 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:53.502005 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:53.502013 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:53.502221 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:53.502601 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:53.502606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:53.502610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:53.503106 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:53.503434 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:53.503728 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 01:14:53.503992 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 01:14:53.505196 13326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:53.605495 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 13119
I20260812 01:14:53.631901 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 13342
Aug 12 01:14:53 dist-test-slave-21b9 krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:14:53.651255 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.632638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.20.151.2:36217)
0812 01:14:53.632858 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:53.632864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:53.632883 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:14:53.633048 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:53.633052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:53.633129 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:53.633214 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:53.634587 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.635437 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.636803 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.637247 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.637358 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.645773 (+ 8415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:53.645792 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:53.645810 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:53.645850 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:53.648785 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:53.649404 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:53.649411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:53.649413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:53.649545 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:53.649842 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:53.649845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:53.649847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:53.650273 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:53.650511 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:53.650784 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 01:14:53.650959 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 01:14:53.651938 13561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:53.747149 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 13342
I20260812 01:14:53.776985 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 13577
Aug 12 01:14:53 dist-test-slave-21b9 krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:14:53.792318 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:53.777900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.20.151.3:33415)
0812 01:14:53.778105 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:53.778111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:53.778129 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:14:53.778229 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:53.778233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:53.778297 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:53.778379 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:53.779687 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.780411 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.781652 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:53.782077 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:53.782190 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:53.786607 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:53.786626 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:53.786643 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:53.786687 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:53.789471 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:53.790170 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:53.790178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:53.790180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:53.790325 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:53.790710 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:53.790714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:53.790717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:53.791102 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:53.791417 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:53.791740 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 01:14:53.791959 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 01:14:53.792977 13771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:53.881409 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 13577
I20260812 01:14:53.908767 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 12975
I20260812 01:14:53.910008 13078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:54.025511 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 12975
I20260812 01:14:54.054706 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46709
--webserver_interface=127.20.151.62
--webserver_port=42617
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:46709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:14:54.406898 13865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:54.407204 13865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:54.407379 13865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:54.415239 13865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:14:54.415395 13865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:54.415483 13865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:54.415570 13865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:14:54.415633 13865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:14:54.425894 13865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:46709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=42617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:54.427417 13865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:54.428896 13865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:54.457157 13872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:54.457180 13873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:54.457157 13876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:54.458458 13865 server_base.cc:1034] running on GCE node
I20260812 01:14:54.459472 13865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:54.461102 13865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:54.462327 13865 hybrid_clock.cc:648] HybridClock initialized: now 1786497294462269 us; error 37 us; skew 500 ppm
I20260812 01:14:54.462775 13865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:54 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497294, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:54.471905 13865 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:14:54.473891 13865 webserver.cc:460] Webserver started at http://127.20.151.62:42617/ using document root <none> and password file <none>
I20260812 01:14:54.474737 13865 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:54.474898 13865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:54.481787 13865 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 01:14:54.485044 13887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:54.486466 13865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:14:54.486660 13865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "2bd08a94670d4f949710c2db4217a847"
format_stamp: "Formatted at 2026-08-12 01:14:51 on dist-test-slave-21b9"
server_key: "dc551acb12969772c7ad90593cb9234f"
server_key_iv: "2b23465d4c26854470b36a0799a7376a"
server_key_version: "encryptionkey@0"
I20260812 01:14:54.487875 13865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:54.508421 13865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:54.513288 13865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:54.513585 13865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:54.533416 13865 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:46709
I20260812 01:14:54.533447 13984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:46709 every 8 connection(s)
I20260812 01:14:54.535066 13865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:14:54.537899 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 13865
I20260812 01:14:54.538731 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:45173
--local_ip_for_outbound_sockets=127.20.151.1
--tserver_master_addrs=127.20.151.62:46709
--webserver_port=37745
--webserver_interface=127.20.151.1
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:14:54.542789 13987 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:14:54.546341 13987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Bootstrap starting.
I20260812 01:14:54.567613 13987 log.cc:824] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:54.580751 13987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:14:54.581342 13987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Bootstrap complete.
I20260812 01:14:54.591553 13987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:54.592504 13987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd08a94670d4f949710c2db4217a847, State: Initialized, Role: FOLLOWER
I20260812 01:14:54.593204 13987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:54.593447 13987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:14:54.593575 13987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:14:54.593755 13987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 1 FOLLOWER]: Advancing to term 2
I20260812 01:14:54.597904 13987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:54.598454 13987 leader_election.cc:302] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bd08a94670d4f949710c2db4217a847; no voters:
I20260812 01:14:54.599118 13987 leader_election.cc:288] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 01:14:54.599335 13993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 01:14:54.600629 13993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 2 LEADER]: Becoming Leader. State: Replica: 2bd08a94670d4f949710c2db4217a847, State: Running, Role: LEADER
I20260812 01:14:54.601248 13993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:54.601686 13987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:14:54.603960 13998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bd08a94670d4f949710c2db4217a847. Latest consensus state: current_term: 2 leader_uuid: "2bd08a94670d4f949710c2db4217a847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } } }
I20260812 01:14:54.604357 13998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:54.604813 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2bd08a94670d4f949710c2db4217a847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } } }
I20260812 01:14:54.605100 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:54.617333 14002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:14:54.630105 14002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:14:54.631693 14002 catalog_manager.cc:1259] Loaded cluster ID: 5a08f33c02ee45dc85c3431cec4e893a
I20260812 01:14:54.631808 14002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:14:54.635073 14002 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 01:14:54.662098 14002 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 01:14:54.684274 14002 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:14:54.688350 14002 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Loaded TSK: 0
I20260812 01:14:54.692780 14002 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 01:14:55.139559 13989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:55.140116 13989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:55.140332 13989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:55.154587 13989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:55.154817 13989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:55.155097 13989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:14:55.173131 13989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:45173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=37745
--tserver_master_addrs=127.20.151.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:55.175751 13989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:55.178367 13989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:55.206979 14036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:55.206902 14033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:55.208405 14038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:55.209728 13989 server_base.cc:1034] running on GCE node
I20260812 01:14:55.210988 13989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:55.212940 13989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:55.214483 13989 hybrid_clock.cc:648] HybridClock initialized: now 1786497295214354 us; error 60 us; skew 500 ppm
I20260812 01:14:55.215150 13989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:55 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497295, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:55.221786 13989 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:14:55.224637 13989 webserver.cc:460] Webserver started at http://127.20.151.1:37745/ using document root <none> and password file <none>
I20260812 01:14:55.225945 13989 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:55.226167 13989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:55.238083 13989 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.004s
I20260812 01:14:55.243618 14046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:55.246785 13989 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260812 01:14:55.247056 13989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "cd60e7746a0447278e642cd1ef1b6d6e"
format_stamp: "Formatted at 2026-08-12 01:14:51 on dist-test-slave-21b9"
server_key: "22967cbb0a5e2a086ccf99c80bf7c8cf"
server_key_iv: "9db4090da819153d639469bef9f7d9d8"
server_key_version: "encryptionkey@0"
I20260812 01:14:55.248919 13989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:55.299005 13989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:55.307897 13989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:55.308442 13989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:55.310325 13989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:55.313253 13989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:55.313424 13989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:55.313645 13989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:55.313772 13989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:55.381119 13989 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:45173
I20260812 01:14:55.381232 14231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:45173 every 8 connection(s)
I20260812 01:14:55.384140 13989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:14:55.386281 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 13989
I20260812 01:14:55.387521 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:36217
--local_ip_for_outbound_sockets=127.20.151.2
--tserver_master_addrs=127.20.151.62:46709
--webserver_port=46473
--webserver_interface=127.20.151.2
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:14:55 dist-test-slave-21b9 krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497295, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:55.438139 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:55.390544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:57651 (local address 127.20.151.62:46709)
0812 01:14:55.392298 (+ 1754us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:55.392343 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:55.394199 (+ 1856us) server_negotiation.cc:407] Connection header received
0812 01:14:55.399364 (+ 5165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:55.399410 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:55.400490 (+ 1080us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:55.401367 (+ 877us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:55.404533 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:55.408294 (+ 3761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:55.412783 (+ 4489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:55.414201 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:55.414480 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:55.422908 (+ 8428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:55.422929 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:55.422957 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:55.423051 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:55.428618 (+ 5567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:55.430710 (+ 2092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:55.430721 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:55.430737 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:55.431038 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:55.432232 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:55.432269 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:55.432274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:55.433303 (+ 1029us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:55.433608 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:55.435772 (+ 2164us) server_negotiation.cc:299] Negotiation successful
0812 01:14:55.436492 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1191,"thread_start_us":741,"threads_started":1}
I20260812 01:14:55.439174 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:55.391496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:57651)
0812 01:14:55.393798 (+ 2302us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:55.393925 (+ 127us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:55.398010 (+ 4085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:55.401850 (+ 3840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:55.401871 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:55.402845 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:55.404117 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:55.404154 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:55.408624 (+ 4470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:55.408629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:55.412436 (+ 3807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:55.412460 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:55.414675 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:55.414694 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:55.415323 (+ 629us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:55.416673 (+ 1350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:55.416748 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:55.422166 (+ 5418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:55.428970 (+ 6804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:55.428984 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:55.429022 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:55.430439 (+ 1417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:55.431324 (+ 885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:55.431330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:55.431334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:55.431958 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:55.433647 (+ 1689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:55.433679 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:55.434694 (+ 1015us) client_negotiation.cc:769] Sending connection context
0812 01:14:55.435442 (+ 748us) client_negotiation.cc:240] Negotiation successful
0812 01:14:55.436763 (+ 1321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1648,"thread_start_us":835,"threads_started":1}
I20260812 01:14:55.444062 14232 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46709
I20260812 01:14:55.444743 14232 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:55.446646 14232 heartbeater.cc:511] Master 127.20.151.62:46709 requested a full tablet report, sending...
I20260812 01:14:55.450990 13919 ts_manager.cc:194] Registered new tserver with Master: cd60e7746a0447278e642cd1ef1b6d6e (127.20.151.1:45173)
I20260812 01:14:55.453925 13919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:57651
W20260812 01:14:56.072103 14247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:56.072657 14247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:56.073000 14247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:56.086913 14247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:56.087116 14247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:56.087496 14247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:14:56.107148 14247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:36217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=46473
--tserver_master_addrs=127.20.151.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:56.110355 14247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:56.113164 14247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:56.136462 14254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:56.136529 14259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:56.136482 14256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:56.139588 14247 server_base.cc:1034] running on GCE node
I20260812 01:14:56.140826 14247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:56.142659 14247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:56.144238 14247 hybrid_clock.cc:648] HybridClock initialized: now 1786497296144102 us; error 101 us; skew 500 ppm
I20260812 01:14:56.144824 14247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:56 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497296, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:56.150714 14247 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:14:56.153328 14247 webserver.cc:460] Webserver started at http://127.20.151.2:46473/ using document root <none> and password file <none>
I20260812 01:14:56.154753 14247 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:56.154991 14247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:56.165645 14247 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 01:14:56.170969 14274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:56.173696 14247 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260812 01:14:56.173975 14247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "8ed72a3b1d5d45988a1bb05914e5595c"
format_stamp: "Formatted at 2026-08-12 01:14:52 on dist-test-slave-21b9"
server_key: "aa9a83d3d0e15a35c8611a7507c3b320"
server_key_iv: "9bd428cb57f7d565ed84b4cc661c0da1"
server_key_version: "encryptionkey@0"
I20260812 01:14:56.175884 14247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:56.246153 14247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:56.254473 14247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:56.255053 14247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:56.257261 14247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:56.259979 14247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:56.260120 14247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:56.260370 14247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:56.260499 14247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:56.323145 14247 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:36217
I20260812 01:14:56.323261 14458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:36217 every 8 connection(s)
I20260812 01:14:56.326511 14247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:14:56.327713 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 14247
I20260812 01:14:56.329339 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:33415
--local_ip_for_outbound_sockets=127.20.151.3
--tserver_master_addrs=127.20.151.62:46709
--webserver_port=34751
--webserver_interface=127.20.151.3
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:14:56 dist-test-slave-21b9 krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497296, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:56.380486 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:56.332024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:44691 (local address 127.20.151.62:46709)
0812 01:14:56.333354 (+ 1330us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:56.333374 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:56.336064 (+ 2690us) server_negotiation.cc:407] Connection header received
0812 01:14:56.341468 (+ 5404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:56.341483 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:56.341666 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:56.341834 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:56.347251 (+ 5417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:56.348788 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:56.354619 (+ 5831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:56.356087 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:56.356387 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:56.365275 (+ 8888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:56.365292 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:56.365295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:56.365379 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:56.371657 (+ 6278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:56.373510 (+ 1853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:56.373516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:56.373519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:56.373806 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:56.374838 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:56.374843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:56.374845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:56.375499 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:56.375756 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:56.377558 (+ 1802us) server_negotiation.cc:299] Negotiation successful
0812 01:14:56.378221 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":448,"threads_started":1}
I20260812 01:14:56.382709 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:56.333514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:44691)
0812 01:14:56.335553 (+ 2039us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:56.335853 (+ 300us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:56.340717 (+ 4864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:56.342723 (+ 2006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:56.342750 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:56.344686 (+ 1936us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:56.346796 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:56.346853 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:56.349239 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:56.349247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:56.354291 (+ 5044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:56.354330 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:56.356577 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:56.356598 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:56.357322 (+ 724us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:56.358679 (+ 1357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:56.358857 (+ 178us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:56.364590 (+ 5733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:56.372010 (+ 7420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:56.372102 (+ 92us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:56.372152 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:56.373268 (+ 1116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:56.374233 (+ 965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:56.374238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:56.374240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:56.374634 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:56.375738 (+ 1104us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:56.375760 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:56.376652 (+ 892us) client_negotiation.cc:769] Sending connection context
0812 01:14:56.377300 (+ 648us) client_negotiation.cc:240] Negotiation successful
0812 01:14:56.378594 (+ 1294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1410,"thread_start_us":837,"threads_started":1}
I20260812 01:14:56.387034 14459 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46709
I20260812 01:14:56.388087 14459 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:56.390687 14459 heartbeater.cc:511] Master 127.20.151.62:46709 requested a full tablet report, sending...
I20260812 01:14:56.395349 13919 ts_manager.cc:194] Registered new tserver with Master: 8ed72a3b1d5d45988a1bb05914e5595c (127.20.151.2:36217)
I20260812 01:14:56.397382 13919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:44691
I20260812 01:14:56.460203 14232 heartbeater.cc:503] Master 127.20.151.62:46709 was elected leader, sending a full tablet report...
W20260812 01:14:57.031904 14471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:57.032526 14471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:57.032805 14471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:57.046473 14471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:57.046720 14471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:57.047091 14471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:14:57.065832 14471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:33415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=34751
--tserver_master_addrs=127.20.151.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:57.068472 14471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:57.071202 14471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:57.097152 14479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:57.097157 14481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:57.097905 14478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:57.098626 14471 server_base.cc:1034] running on GCE node
I20260812 01:14:57.099987 14471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:57.101886 14471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:57.103472 14471 hybrid_clock.cc:648] HybridClock initialized: now 1786497297103327 us; error 97 us; skew 500 ppm
I20260812 01:14:57.104071 14471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:57 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497297, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:57.110309 14471 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:14:57.113108 14471 webserver.cc:460] Webserver started at http://127.20.151.3:34751/ using document root <none> and password file <none>
I20260812 01:14:57.114465 14471 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:57.114696 14471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:57.125758 14471 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 01:14:57.131451 14492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:57.134287 14471 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260812 01:14:57.134603 14471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "1fc0fa08bf1d493887cde9461b7c8a30"
format_stamp: "Formatted at 2026-08-12 01:14:52 on dist-test-slave-21b9"
server_key: "f31276f95b0774db6843b3ea7dda32ef"
server_key_iv: "6c9c51389638d76bed03939fbf9d7568"
server_key_version: "encryptionkey@0"
I20260812 01:14:57.136559 14471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:57.182158 14471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:57.190675 14471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:57.191263 14471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:57.193312 14471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:57.195578 14471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:57.195698 14471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:57.195825 14471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:57.195910 14471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:57.230746 14471 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:33415
I20260812 01:14:57.230806 14674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:33415 every 8 connection(s)
I20260812 01:14:57.232337 14471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:14:57.233466 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 14471
Aug 12 01:14:57 dist-test-slave-21b9 krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497297, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:14:57.250679 14683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.234501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.20.151.62:46709)
0812 01:14:57.235643 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.235648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.235665 (+ 17us) server_negotiation.cc:407] Connection header received
0812 01:14:57.235745 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.235748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.235811 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.235897 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:57.238316 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.239154 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.240572 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.241274 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.241424 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.244596 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:57.244602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:57.244605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:57.244638 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:57.247623 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:57.248528 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:57.248533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:57.248535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:57.248696 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:57.249131 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:57.249135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:57.249137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:57.249450 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:57.249666 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.250077 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.250316 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":916,"threads_started":1}
I20260812 01:14:57.256625 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.235755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:41987 (local address 127.20.151.62:46709)
0812 01:14:57.236250 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.236254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.239050 (+ 2796us) server_negotiation.cc:407] Connection header received
0812 01:14:57.241060 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.241067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.241164 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.241282 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:57.243438 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.244314 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.245996 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.246623 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.246766 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.251007 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:57.251014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:57.251019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:57.251064 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:57.253902 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:57.254613 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:57.254616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:57.254618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:57.254723 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:57.255103 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:57.255105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:57.255107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:57.255392 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:57.255515 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.256164 (+ 649us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.256341 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":268,"threads_started":1}
I20260812 01:14:57.257746 14686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.235697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:41987)
0812 01:14:57.238880 (+ 3183us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:57.238936 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:57.240755 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:57.241555 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:57.241569 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:57.242307 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:14:57.243176 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.243205 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.244479 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.244484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.245829 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.245854 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.246873 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.246878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.247111 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.247789 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:14:57.247820 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:14:57.250696 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:14:57.254030 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:57.254038 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:57.254052 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:57.254522 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:57.254816 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:14:57.254820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:14:57.254822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:14:57.255020 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:14:57.255515 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:14:57.255521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:14:57.255842 (+ 321us) client_negotiation.cc:769] Sending connection context
0812 01:14:57.256073 (+ 231us) client_negotiation.cc:240] Negotiation successful
0812 01:14:57.256550 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2872,"thread_start_us":2649,"threads_started":1}
I20260812 01:14:57.259069 14679 heartbeater.cc:347] Connected to a master server at 127.20.151.62:46709
I20260812 01:14:57.259392 14679 heartbeater.cc:464] Registering TS with master...
I20260812 01:14:57.260147 14679 heartbeater.cc:511] Master 127.20.151.62:46709 requested a full tablet report, sending...
I20260812 01:14:57.261592 13919 ts_manager.cc:194] Registered new tserver with Master: 1fc0fa08bf1d493887cde9461b7c8a30 (127.20.151.3:33415)
I20260812 01:14:57.262519 13919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:41987
I20260812 01:14:57.265698 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:14:57.272617 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.266772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.20.151.62:46709)
0812 01:14:57.267023 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.267028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.267334 (+ 306us) server_negotiation.cc:407] Connection header received
0812 01:14:57.267553 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.267557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.267696 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.267819 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:14:57.268241 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.268942 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.270384 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.270779 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.270889 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.271453 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:14:57.271992 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:14:57.272135 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.272239 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.272323 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260812 01:14:57.273638 13919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:14:57.281205 13919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:14:57.322016 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.294505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.20.151.2:36217)
0812 01:14:57.313563 (+ 19058us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.313570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.313599 (+ 29us) server_negotiation.cc:407] Connection header received
0812 01:14:57.313732 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.313739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.313991 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.314197 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:57.315683 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.317185 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.319576 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.320802 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.320934 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.321168 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.321479 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.321657 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18900,"thread_start_us":18764,"threads_started":1}
I20260812 01:14:57.322152 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.294213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:36217 (local address 127.0.0.1:51644)
0812 01:14:57.312002 (+ 17789us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:57.312031 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:57.312190 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:57.314198 (+ 2008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:57.314202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:57.314230 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:57.315446 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.315478 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.317338 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.317345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.319370 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.319387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.321030 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.321037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.321237 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.321279 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 01:14:57.321386 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 01:14:57.321618 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17624,"mutex_wait_us":79,"thread_start_us":17442,"threads_started":1}
I20260812 01:14:57.322780 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.294258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.20.151.3:33415)
0812 01:14:57.314080 (+ 19822us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.314088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.314121 (+ 33us) server_negotiation.cc:407] Connection header received
0812 01:14:57.314212 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.314219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.314459 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.314666 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:57.315312 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.316990 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.319658 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.321061 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.321192 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.321500 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.322260 (+ 760us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.322445 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19656,"thread_start_us":19482,"threads_started":1}
I20260812 01:14:57.323335 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.294050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:33415 (local address 127.0.0.1:58532)
0812 01:14:57.309558 (+ 15508us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:57.309614 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:57.309844 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:57.314648 (+ 4804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:57.314651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:57.314669 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:57.315088 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.315100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.317142 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.317146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.319487 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.319499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.321670 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.321677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.321856 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.321890 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 01:14:57.321975 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 01:14:57.322114 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15210,"mutex_wait_us":3,"thread_start_us":15008,"threads_started":1}
I20260812 01:14:57.325815 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.296603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:45173 (local address 127.0.0.1:43534)
0812 01:14:57.312415 (+ 15812us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:57.312435 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:57.312572 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:57.314080 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:57.314088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:57.314124 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:57.314808 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.314826 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.316701 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.316707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.319370 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.319390 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.325007 (+ 5617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.325014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.325187 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.325218 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 01:14:57.325307 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 01:14:57.325440 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15678,"mutex_wait_us":80,"thread_start_us":15513,"threads_started":1}
I20260812 01:14:57.326714 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.296944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.20.151.1:45173)
0812 01:14:57.313170 (+ 16226us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.313180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.313216 (+ 36us) server_negotiation.cc:407] Connection header received
0812 01:14:57.313334 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.313344 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.313661 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.313880 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:57.315028 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.316559 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.319556 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.321057 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.324622 (+ 3565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.324865 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.326159 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.326346 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16039,"thread_start_us":15885,"threads_started":1}
I20260812 01:14:57.327549 14342 tablet_service.cc:1467] Processing CreateTablet for tablet dc5cac5346ad4766a315231af10a29de (DEFAULT_TABLE table=test-table [id=548c1b2fce264fea92a5a7a7be6359d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:14:57.329128 14342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5cac5346ad4766a315231af10a29de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:57.330235 14127 tablet_service.cc:1467] Processing CreateTablet for tablet dc5cac5346ad4766a315231af10a29de (DEFAULT_TABLE table=test-table [id=548c1b2fce264fea92a5a7a7be6359d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:14:57.326997 14568 tablet_service.cc:1467] Processing CreateTablet for tablet dc5cac5346ad4766a315231af10a29de (DEFAULT_TABLE table=test-table [id=548c1b2fce264fea92a5a7a7be6359d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:14:57.331439 14127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5cac5346ad4766a315231af10a29de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:57.332197 14568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5cac5346ad4766a315231af10a29de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:14:57.343943 14705 tablet_bootstrap.cc:492] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: Bootstrap starting.
I20260812 01:14:57.345999 14708 tablet_bootstrap.cc:492] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c: Bootstrap starting.
I20260812 01:14:57.347527 14709 tablet_bootstrap.cc:492] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30: Bootstrap starting.
I20260812 01:14:57.347983 14705 tablet_bootstrap.cc:654] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:57.349398 14705 log.cc:824] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:57.349538 14708 tablet_bootstrap.cc:654] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:57.350346 14709 tablet_bootstrap.cc:654] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30: Neither blocks nor log segments found. Creating new log.
I20260812 01:14:57.350832 14708 log.cc:824] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:57.351562 14709 log.cc:824] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:57.353969 14705 tablet_bootstrap.cc:492] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: No bootstrap required, opened a new log
I20260812 01:14:57.354234 14705 ts_tablet_manager.cc:1397] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260812 01:14:57.354831 14708 tablet_bootstrap.cc:492] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c: No bootstrap required, opened a new log
I20260812 01:14:57.355109 14708 ts_tablet_manager.cc:1397] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260812 01:14:57.355140 14709 tablet_bootstrap.cc:492] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30: No bootstrap required, opened a new log
I20260812 01:14:57.355408 14709 ts_tablet_manager.cc:1397] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 01:14:57.361739 14705 raft_consensus.cc:348] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:57.362242 14705 raft_consensus.cc:374] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:57.362346 14705 raft_consensus.cc:729] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd60e7746a0447278e642cd1ef1b6d6e, State: Initialized, Role: FOLLOWER
I20260812 01:14:57.363015 14705 consensus_queue.cc:260] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:57.365574 14708 raft_consensus.cc:348] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:57.365574 14709 raft_consensus.cc:348] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:57.365998 14705 ts_tablet_manager.cc:1428] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260812 01:14:57.366019 14709 raft_consensus.cc:374] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:57.366016 14708 raft_consensus.cc:374] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:14:57.366117 14709 raft_consensus.cc:729] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc0fa08bf1d493887cde9461b7c8a30, State: Initialized, Role: FOLLOWER
I20260812 01:14:57.366117 14708 raft_consensus.cc:729] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed72a3b1d5d45988a1bb05914e5595c, State: Initialized, Role: FOLLOWER
I20260812 01:14:57.366796 14708 consensus_queue.cc:260] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:57.366796 14709 consensus_queue.cc:260] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:57.367961 14459 heartbeater.cc:503] Master 127.20.151.62:46709 was elected leader, sending a full tablet report...
I20260812 01:14:57.368659 14708 ts_tablet_manager.cc:1428] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260812 01:14:57.369892 14679 heartbeater.cc:503] Master 127.20.151.62:46709 was elected leader, sending a full tablet report...
I20260812 01:14:57.370561 14709 ts_tablet_manager.cc:1428] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260812 01:14:57.375547 14714 raft_consensus.cc:482] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:14:57.375794 14714 raft_consensus.cc:504] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:57.377440 14714 leader_election.cc:288] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fc0fa08bf1d493887cde9461b7c8a30 (127.20.151.3:33415), 8ed72a3b1d5d45988a1bb05914e5595c (127.20.151.2:36217)
I20260812 01:14:57.384927 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.377889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:58569 (local address 127.20.151.2:36217)
0812 01:14:57.378099 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.378106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.378966 (+ 860us) server_negotiation.cc:407] Connection header received
0812 01:14:57.379186 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.379190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.379248 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.379347 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:57.379914 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.380647 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.382941 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.383911 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.384072 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.384164 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.384557 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.384687 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:14:57.385321 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.377744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:36217 (local address 127.20.151.1:58569)
0812 01:14:57.378852 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:57.378874 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:57.379014 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:57.379374 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:57.379379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:57.379403 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:57.379780 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.379790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.380785 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.380794 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.382798 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.382811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.384180 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.384188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.384394 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.384420 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 01:14:57.384514 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 01:14:57.384651 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":33,"thread_start_us":946,"threads_started":1}
I20260812 01:14:57.387163 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.377901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:45475 (local address 127.20.151.3:33415)
0812 01:14:57.378288 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.378295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.379100 (+ 805us) server_negotiation.cc:407] Connection header received
0812 01:14:57.379261 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.379291 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.379360 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.379435 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:14:57.381041 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.382079 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.384666 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.385888 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.386017 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.386102 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.386780 (+ 678us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.386922 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260812 01:14:57.387255 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.377894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:33415 (local address 127.20.151.1:45475)
0812 01:14:57.379010 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:57.379024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 01:14:57.379131 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:14:57.380423 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:14:57.380428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:14:57.380455 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:14:57.380858 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.380869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.382222 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.382228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.384496 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:14:57.384512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.386428 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.386434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:14:57.386612 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.386644 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:14:57.386727 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 01:14:57.386863 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":26,"thread_start_us":390,"threads_started":1}
I20260812 01:14:57.387050 14380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5cac5346ad4766a315231af10a29de" candidate_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" is_pre_election: true
I20260812 01:14:57.387701 14380 raft_consensus.cc:2454] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd60e7746a0447278e642cd1ef1b6d6e in term 0.
I20260812 01:14:57.388437 14047 leader_election.cc:302] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed72a3b1d5d45988a1bb05914e5595c, cd60e7746a0447278e642cd1ef1b6d6e; no voters:
I20260812 01:14:57.388533 14601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5cac5346ad4766a315231af10a29de" candidate_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" is_pre_election: true
I20260812 01:14:57.389053 14601 raft_consensus.cc:2454] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd60e7746a0447278e642cd1ef1b6d6e in term 0.
W20260812 01:14:57.389277 14236 tablet.cc:2369] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:14:57.389516 14714 raft_consensus.cc:2781] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:14:57.389683 14714 raft_consensus.cc:482] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:14:57.389797 14714 raft_consensus.cc:3037] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:57.392359 14714 raft_consensus.cc:504] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:57.393096 14714 leader_election.cc:288] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [CANDIDATE]: Term 1 election: Requested vote from peers 1fc0fa08bf1d493887cde9461b7c8a30 (127.20.151.3:33415), 8ed72a3b1d5d45988a1bb05914e5595c (127.20.151.2:36217)
I20260812 01:14:57.393545 14601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5cac5346ad4766a315231af10a29de" candidate_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc0fa08bf1d493887cde9461b7c8a30"
I20260812 01:14:57.393622 14380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5cac5346ad4766a315231af10a29de" candidate_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed72a3b1d5d45988a1bb05914e5595c"
I20260812 01:14:57.393772 14601 raft_consensus.cc:3037] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:57.393832 14380 raft_consensus.cc:3037] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:14:57.397105 14601 raft_consensus.cc:2454] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd60e7746a0447278e642cd1ef1b6d6e in term 1.
I20260812 01:14:57.397105 14380 raft_consensus.cc:2454] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd60e7746a0447278e642cd1ef1b6d6e in term 1.
I20260812 01:14:57.397572 14047 leader_election.cc:302] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed72a3b1d5d45988a1bb05914e5595c, cd60e7746a0447278e642cd1ef1b6d6e; no voters:
I20260812 01:14:57.397909 14714 raft_consensus.cc:2781] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:14:57.398375 14714 raft_consensus.cc:686] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 1 LEADER]: Becoming Leader. State: Replica: cd60e7746a0447278e642cd1ef1b6d6e, State: Running, Role: LEADER
I20260812 01:14:57.398934 14714 consensus_queue.cc:237] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } }
I20260812 01:14:57.403756 13919 catalog_manager.cc:5495] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e reported cstate change: term changed from 0 to 1, leader changed from <none> to cd60e7746a0447278e642cd1ef1b6d6e (127.20.151.1). New cstate: current_term: 1 leader_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd60e7746a0447278e642cd1ef1b6d6e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 45173 } health_report { overall_health: HEALTHY } } }
I20260812 01:14:57.424566 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.418793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.20.151.1:45173)
0812 01:14:57.418982 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.418986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.419008 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:14:57.419348 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.419353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.419454 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.419591 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:14:57.419974 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.420730 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.422266 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.422720 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.422821 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.423380 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:14:57.423934 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:14:57.424070 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.424241 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.424323 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 01:14:57.433262 14380 raft_consensus.cc:1261] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c [term 1 FOLLOWER]: Refusing update from remote peer cd60e7746a0447278e642cd1ef1b6d6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:14:57.433264 14601 raft_consensus.cc:1261] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30 [term 1 FOLLOWER]: Refusing update from remote peer cd60e7746a0447278e642cd1ef1b6d6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:14:57.434288 14723 consensus_queue.cc:1035] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ed72a3b1d5d45988a1bb05914e5595c" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:14:57.434677 14714 consensus_queue.cc:1035] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc0fa08bf1d493887cde9461b7c8a30" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:14:57.461050 13919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:14:57.461328 13919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:14:57.465109 13919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Sending DeleteTablet for 3 replicas of tablet dc5cac5346ad4766a315231af10a29de
I20260812 01:14:57.466400 14127 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5cac5346ad4766a315231af10a29de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:14:57 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:43534
I20260812 01:14:57.466391 14342 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5cac5346ad4766a315231af10a29de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:14:57 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:51644
I20260812 01:14:57.466403 14568 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5cac5346ad4766a315231af10a29de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:14:57 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:58532
I20260812 01:14:57.467965 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 13989
I20260812 01:14:57.479391 14745 tablet_replica.cc:330] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: stopping tablet replica
I20260812 01:14:57.479631 14744 tablet_replica.cc:330] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c: stopping tablet replica
I20260812 01:14:57.479933 14745 raft_consensus.cc:2229] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:14:57.480101 14744 raft_consensus.cc:2229] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:14:57.480698 14744 raft_consensus.cc:2258] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:14:57.480705 14745 raft_consensus.cc:2258] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:14:57.483350 14745 ts_tablet_manager.cc:1905] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:14:57.483923 14743 tablet_replica.cc:330] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30: stopping tablet replica
I20260812 01:14:57.483224 14744 ts_tablet_manager.cc:1905] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:14:57.491469 14745 ts_tablet_manager.cc:1918] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:14:57.491681 14745 log.cc:1196] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/dc5cac5346ad4766a315231af10a29de
I20260812 01:14:57.491542 14744 ts_tablet_manager.cc:1918] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:14:57.492275 14744 log.cc:1196] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/dc5cac5346ad4766a315231af10a29de
I20260812 01:14:57.492295 14745 ts_tablet_manager.cc:1939] T dc5cac5346ad4766a315231af10a29de P cd60e7746a0447278e642cd1ef1b6d6e: Deleting consensus metadata
I20260812 01:14:57.492908 14744 ts_tablet_manager.cc:1939] T dc5cac5346ad4766a315231af10a29de P 8ed72a3b1d5d45988a1bb05914e5595c: Deleting consensus metadata
I20260812 01:14:57.493430 14743 raft_consensus.cc:2229] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:14:57.494019 14743 raft_consensus.cc:2258] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:14:57.494376 13893 catalog_manager.cc:4841] TS cd60e7746a0447278e642cd1ef1b6d6e (127.20.151.1:45173): tablet dc5cac5346ad4766a315231af10a29de (table test-table [id=548c1b2fce264fea92a5a7a7be6359d6]) successfully deleted
I20260812 01:14:57.494931 13888 catalog_manager.cc:4841] TS 8ed72a3b1d5d45988a1bb05914e5595c (127.20.151.2:36217): tablet dc5cac5346ad4766a315231af10a29de (table test-table [id=548c1b2fce264fea92a5a7a7be6359d6]) successfully deleted
I20260812 01:14:57.495494 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.469180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.20.151.1:45173)
0812 01:14:57.469358 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.469364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.469382 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:14:57.469440 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.469446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.469528 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.469628 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:57.471738 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.473007 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.475035 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.475773 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.475902 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.482872 (+ 6970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:57.482889 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:57.482908 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:57.482952 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:57.489037 (+ 6085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:57.490291 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:57.490301 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:57.490304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:57.490497 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:57.490991 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:57.490996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:57.491000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:57.491533 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:57.494405 (+ 2872us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.494573 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.494832 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:14:57.496194 14223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:57.497177 14743 ts_tablet_manager.cc:1905] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:14:57.503214 14743 ts_tablet_manager.cc:1918] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:14:57.503401 14743 log.cc:1196] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/dc5cac5346ad4766a315231af10a29de
I20260812 01:14:57.503857 14743 ts_tablet_manager.cc:1939] T dc5cac5346ad4766a315231af10a29de P 1fc0fa08bf1d493887cde9461b7c8a30: Deleting consensus metadata
I20260812 01:14:57.505513 13892 catalog_manager.cc:4841] TS 1fc0fa08bf1d493887cde9461b7c8a30 (127.20.151.3:33415): tablet dc5cac5346ad4766a315231af10a29de (table test-table [id=548c1b2fce264fea92a5a7a7be6359d6]) successfully deleted
I20260812 01:14:57.596418 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 13989
I20260812 01:14:57.626230 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 14247
I20260812 01:14:57.640986 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.627093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.20.151.2:36217)
0812 01:14:57.627327 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.627333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.627355 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:14:57.627518 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.627522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.627600 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.627684 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:57.629161 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.630023 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.631704 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.632113 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.632209 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.635255 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:57.635305 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:57.635330 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:57.635375 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:57.638047 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:57.638871 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:57.638879 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:57.638881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:57.639031 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:57.639512 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:57.639515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:57.639517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:57.639931 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:57.640173 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.640474 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.640656 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 01:14:57.641743 14454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:57.732542 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 14247
I20260812 01:14:57.758988 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 14471
I20260812 01:14:57.772554 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:57.759835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.20.151.3:33415)
0812 01:14:57.760056 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 01:14:57.760063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:14:57.760084 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:14:57.760272 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:14:57.760276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:14:57.760342 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:14:57.760456 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:14:57.761854 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.762595 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.763981 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:14:57.764374 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:14:57.764484 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:14:57.767234 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:14:57.767251 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:14:57.767305 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:14:57.767345 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:14:57.769775 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:57.770472 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:57.770481 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:57.770484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:57.770630 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:14:57.770991 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:14:57.770996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:14:57.770999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:14:57.771420 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:14:57.771716 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0812 01:14:57.772008 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 01:14:57.772234 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 01:14:57.773185 14668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:57.862519 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 14471
I20260812 01:14:57.889503 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 13865
I20260812 01:14:57.890673 13975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:14:58.007258 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 13865
I20260812 01:14:58.041230 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46709
--webserver_interface=127.20.151.62
--webserver_port=42617
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:46709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:14:58.390198 14754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:58.390532 14754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:58.390678 14754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:58.397998 14754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:14:58.398125 14754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:58.398209 14754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:58.398288 14754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:14:58.398350 14754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:14:58.408376 14754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:46709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:46709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=42617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:58.409763 14754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:58.411135 14754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:58.423491 14765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:58.423456 14769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:58.423566 14764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:58.424319 14754 server_base.cc:1034] running on GCE node
I20260812 01:14:58.425165 14754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:58.426683 14754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:58.427913 14754 hybrid_clock.cc:648] HybridClock initialized: now 1786497298427854 us; error 43 us; skew 500 ppm
I20260812 01:14:58.428316 14754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:58 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497298, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:58.431793 14754 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:14:58.433339 14754 webserver.cc:460] Webserver started at http://127.20.151.62:42617/ using document root <none> and password file <none>
I20260812 01:14:58.434072 14754 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:58.434190 14754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:58.439940 14754 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 01:14:58.442993 14784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:58.444383 14754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:14:58.444597 14754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "2bd08a94670d4f949710c2db4217a847"
format_stamp: "Formatted at 2026-08-12 01:14:51 on dist-test-slave-21b9"
server_key: "dc551acb12969772c7ad90593cb9234f"
server_key_iv: "2b23465d4c26854470b36a0799a7376a"
server_key_version: "encryptionkey@0"
I20260812 01:14:58.445649 14754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:58.466616 14754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:58.470996 14754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:58.471261 14754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:58.490305 14754 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:46709
I20260812 01:14:58.490329 14868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:46709 every 8 connection(s)
I20260812 01:14:58.491952 14754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:14:58.499168 14869 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:14:58.502573 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 14754
I20260812 01:14:58.502857 14869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Bootstrap starting.
I20260812 01:14:58.503679 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:45173
--local_ip_for_outbound_sockets=127.20.151.1
--tserver_master_addrs=127.20.151.62:46709
--webserver_port=37745
--webserver_interface=127.20.151.1
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:14:58.527539 14869 log.cc:824] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Log is configured to *not* fsync() on all Append() calls
I20260812 01:14:58.544662 14869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:14:58.545238 14869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847: Bootstrap complete.
I20260812 01:14:58.555442 14869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:58.556387 14869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd08a94670d4f949710c2db4217a847, State: Initialized, Role: FOLLOWER
I20260812 01:14:58.557008 14869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:58.557240 14869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:14:58.557354 14869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:14:58.557510 14869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 2 FOLLOWER]: Advancing to term 3
I20260812 01:14:58.561511 14869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:58.562013 14869 leader_election.cc:302] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bd08a94670d4f949710c2db4217a847; no voters:
I20260812 01:14:58.562698 14869 leader_election.cc:288] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 01:14:58.562844 14880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 01:14:58.564002 14880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [term 3 LEADER]: Becoming Leader. State: Replica: 2bd08a94670d4f949710c2db4217a847, State: Running, Role: LEADER
I20260812 01:14:58.564661 14880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } }
I20260812 01:14:58.564976 14869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:14:58.567421 14882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bd08a94670d4f949710c2db4217a847. Latest consensus state: current_term: 3 leader_uuid: "2bd08a94670d4f949710c2db4217a847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } } }
I20260812 01:14:58.567807 14882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:58.568368 14881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2bd08a94670d4f949710c2db4217a847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd08a94670d4f949710c2db4217a847" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 46709 } } }
I20260812 01:14:58.568675 14881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd08a94670d4f949710c2db4217a847 [sys.catalog]: This master's current role is: LEADER
I20260812 01:14:58.578703 14888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:14:58.587844 14888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:14:58.589457 14888 catalog_manager.cc:1259] Loaded cluster ID: 5a08f33c02ee45dc85c3431cec4e893a
I20260812 01:14:58.589648 14888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 01:14:58.591794 14888 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f39a53559ad google::LogMessage::Fail() at ??:0
@ 0x7f39a535888a google::LogMessage::SendToLog() at ??:0
@ 0x7f39a53554de google::LogMessage::Flush() at ??:0
@ 0x7f39a5356a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f39bad88512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f39bad85976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f39bae0d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f39bae0d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f39a7fa1345 kudu::ThreadPool::DispatchThread() at ??:0
I20260812 01:14:58.641687 14907 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f39a7fa491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f39a7fa4799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f39a7f7feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f39ab2596db start_thread at ??:0
@ 0x7f39a41ef61f clone at ??:0
W20260812 01:14:58.946631 14876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:58.946945 14876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:58.947059 14876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:58.954741 14876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:58.954892 14876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:58.955010 14876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:14:58.965204 14876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:45173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=37745
--tserver_master_addrs=127.20.151.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:58.966540 14876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:58.967952 14876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:58.981287 14931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:58.981338 14935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:58.981315 14932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:58.982259 14876 server_base.cc:1034] running on GCE node
I20260812 01:14:58.982897 14876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:58.983947 14876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:58.985147 14876 hybrid_clock.cc:648] HybridClock initialized: now 1786497298985086 us; error 55 us; skew 500 ppm
I20260812 01:14:58.985460 14876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:58 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497298, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:58.988866 14876 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:14:58.990435 14876 webserver.cc:460] Webserver started at http://127.20.151.1:37745/ using document root <none> and password file <none>
I20260812 01:14:58.991142 14876 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:58.991292 14876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:58.996907 14876 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 01:14:58.999699 14946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:59.001149 14876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:14:59.001325 14876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "cd60e7746a0447278e642cd1ef1b6d6e"
format_stamp: "Formatted at 2026-08-12 01:14:51 on dist-test-slave-21b9"
server_key: "22967cbb0a5e2a086ccf99c80bf7c8cf"
server_key_iv: "9db4090da819153d639469bef9f7d9d8"
server_key_version: "encryptionkey@0"
I20260812 01:14:59.002355 14876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:59.038074 14876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:59.042572 14876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:59.042848 14876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:59.043936 14876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:59.045370 14876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:59.045488 14876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:59.045611 14876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:59.045686 14876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:59.077670 14876 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:45173
I20260812 01:14:59.077720 15138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:45173 every 8 connection(s)
I20260812 01:14:59.079144 14876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:14:59.081810 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 14876
I20260812 01:14:59.082471 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:36217
--local_ip_for_outbound_sockets=127.20.151.2
--tserver_master_addrs=127.20.151.62:46709
--webserver_port=46473
--webserver_interface=127.20.151.2
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:14:59.084468 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:59.082170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:41561)
0812 01:14:59.083012 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:59.083327 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":304,"threads_started":1}
W20260812 01:14:59.085237 15140 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:46709 (0 consecutive failures): Network error: Failed to ping master at 127.20.151.62:46709: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
I20260812 01:14:59.086131 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:59.085746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:51313)
0812 01:14:59.085907 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:59.085991 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 01:14:59.087064 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:59.086694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:48195)
0812 01:14:59.086831 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:59.086918 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 01:14:59.087333 15140 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 01:14:59.433005 15143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:59.433320 15143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:59.433434 15143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:59.440935 15143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:59.441074 15143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:59.441216 15143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:14:59.451404 15143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:36217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=46473
--tserver_master_addrs=127.20.151.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:59.452735 15143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:59.454138 15143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:59.468163 15151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:59.468277 15148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:59.468253 15155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:59.469134 15143 server_base.cc:1034] running on GCE node
I20260812 01:14:59.469861 15143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:59.470912 15143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:59.472131 15143 hybrid_clock.cc:648] HybridClock initialized: now 1786497299472077 us; error 39 us; skew 500 ppm
I20260812 01:14:59.472465 15143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:59 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497299, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:59.479379 15143 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:14:59.481030 15143 webserver.cc:460] Webserver started at http://127.20.151.2:46473/ using document root <none> and password file <none>
I20260812 01:14:59.481750 15143 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:59.481884 15143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:59.487671 15143 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 01:14:59.490717 15169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:59.492167 15143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:14:59.492344 15143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "8ed72a3b1d5d45988a1bb05914e5595c"
format_stamp: "Formatted at 2026-08-12 01:14:52 on dist-test-slave-21b9"
server_key: "aa9a83d3d0e15a35c8611a7507c3b320"
server_key_iv: "9bd428cb57f7d565ed84b4cc661c0da1"
server_key_version: "encryptionkey@0"
I20260812 01:14:59.493374 15143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:59.517988 15143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:59.522451 15143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:59.522759 15143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:59.523960 15143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:14:59.525520 15143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:14:59.525636 15143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:59.525758 15143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:14:59.525839 15143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:59.561090 15143 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:36217
I20260812 01:14:59.561160 15356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:36217 every 8 connection(s)
I20260812 01:14:59.562688 15143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:14:59.566447 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 15143
I20260812 01:14:59.567075 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:33415
--local_ip_for_outbound_sockets=127.20.151.3
--tserver_master_addrs=127.20.151.62:46709
--webserver_port=34751
--webserver_interface=127.20.151.3
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:14:59.568102 15362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:59.565810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:48887)
0812 01:14:59.566763 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:59.567116 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":378,"threads_started":1}
W20260812 01:14:59.568823 15357 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:46709 (0 consecutive failures): Network error: Failed to ping master at 127.20.151.62:46709: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
I20260812 01:14:59.569797 15362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:59.569400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:52883)
0812 01:14:59.569595 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:59.569676 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 01:14:59.570724 15362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:14:59.570379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:45955)
0812 01:14:59.570530 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 01:14:59.570612 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 01:14:59.570958 15357 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 01:14:59.920457 15363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:14:59.920769 15363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:14:59.920879 15363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:14:59.928347 15363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:14:59.928469 15363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:14:59.928599 15363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:14:59.938573 15363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:33415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=34751
--tserver_master_addrs=127.20.151.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:14:59.939934 15363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:14:59.941239 15363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:14:59.954407 15372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:59.954388 15373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:14:59.954497 15376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:14:59.955575 15363 server_base.cc:1034] running on GCE node
I20260812 01:14:59.956198 15363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:14:59.957140 15363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:14:59.958339 15363 hybrid_clock.cc:648] HybridClock initialized: now 1786497299958280 us; error 49 us; skew 500 ppm
I20260812 01:14:59.958655 15363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:14:59 dist-test-slave-21b9 krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497299, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:14:59.961930 15363 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:14:59.963555 15363 webserver.cc:460] Webserver started at http://127.20.151.3:34751/ using document root <none> and password file <none>
I20260812 01:14:59.964262 15363 fs_manager.cc:359] Metadata directory not provided
I20260812 01:14:59.964389 15363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:14:59.969920 15363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 01:14:59.973022 15384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:14:59.974480 15363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:14:59.974646 15363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "1fc0fa08bf1d493887cde9461b7c8a30"
format_stamp: "Formatted at 2026-08-12 01:14:52 on dist-test-slave-21b9"
server_key: "f31276f95b0774db6843b3ea7dda32ef"
server_key_iv: "6c9c51389638d76bed03939fbf9d7568"
server_key_version: "encryptionkey@0"
I20260812 01:14:59.975746 15363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:14:59.993798 15363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:14:59.998348 15363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:14:59.998617 15363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:14:59.999730 15363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:15:00.001381 15363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:15:00.001483 15363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:00.001601 15363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:15:00.001682 15363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:00.037400 15363 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:33415
I20260812 01:15:00.037472 15579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:33415 every 8 connection(s)
I20260812 01:15:00.038918 15363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:15:00.041651 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 15363
I20260812 01:15:00.044492 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:00.042303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:32771)
0812 01:15:00.043149 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:00.043585 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":333,"threads_started":1}
W20260812 01:15:00.045248 15580 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:46709 (0 consecutive failures): Network error: Failed to ping master at 127.20.151.62:46709: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
I20260812 01:15:00.046099 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:00.045757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:52425)
0812 01:15:00.045916 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:00.045987 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 01:15:00.046936 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:00.046603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:33477)
0812 01:15:00.046747 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:00.046824 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 01:15:00.047176 15580 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 01:15:00.089035 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:00.088096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:34977)
0812 01:15:00.088648 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:00.088761 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":268,"threads_started":1}
I20260812 01:15:00.572384 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:00.571527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:58821)
0812 01:15:00.572030 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:00.572127 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":244,"threads_started":1}
I20260812 01:15:01.048549 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:01.047698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:38315)
0812 01:15:01.048175 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:01.048295 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":228,"threads_started":1}
I20260812 01:15:01.091027 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:01.090104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:59471)
0812 01:15:01.090735 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:01.090833 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":322,"threads_started":1}
I20260812 01:15:01.574122 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:01.573312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:54345)
0812 01:15:01.573853 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:01.573942 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":278,"threads_started":1}
I20260812 01:15:02.050180 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:02.049427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:49713)
0812 01:15:02.049917 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:02.050002 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
I20260812 01:15:02.093026 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:02.092072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:58389)
0812 01:15:02.092732 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:02.092827 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":377,"threads_started":1}
I20260812 01:15:02.576007 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:02.575088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:60331)
0812 01:15:02.575648 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:02.575765 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":309,"threads_started":1}
I20260812 01:15:03.051981 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:03.051170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:39641)
0812 01:15:03.051692 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:03.051786 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":296,"threads_started":1}
I20260812 01:15:03.094766 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:03.093983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:42003)
0812 01:15:03.094475 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:03.094572 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":255,"threads_started":1}
I20260812 01:15:03.577931 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:03.577007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:42497)
0812 01:15:03.577652 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:03.577736 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":281,"threads_started":1}
I20260812 01:15:04.053704 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:04.052872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:36183)
0812 01:15:04.053371 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:04.053455 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":270,"threads_started":1}
I20260812 01:15:04.096645 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:04.095758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:48161)
0812 01:15:04.096293 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:04.096387 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":289,"threads_started":1}
I20260812 01:15:04.579746 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:04.578842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:38679)
0812 01:15:04.579448 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:04.579549 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":294,"threads_started":1}
I20260812 01:15:05.055538 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:05.054718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:52983)
0812 01:15:05.055225 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:05.055351 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":250,"threads_started":1}
I20260812 01:15:05.098426 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:05.097588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:42371)
0812 01:15:05.098112 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:05.098215 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":270,"threads_started":1}
I20260812 01:15:05.581405 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:05.580677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:55483)
0812 01:15:05.581144 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:05.581236 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":236,"threads_started":1}
I20260812 01:15:06.057396 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:06.056629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:42125)
0812 01:15:06.057118 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:06.057223 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":236,"threads_started":1}
I20260812 01:15:06.100394 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:06.099544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:34021)
0812 01:15:06.100083 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:06.100180 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":230,"threads_started":1}
I20260812 01:15:06.583243 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:06.582375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:45247)
0812 01:15:06.582918 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:06.583009 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":235,"threads_started":1}
I20260812 01:15:07.059250 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:07.058393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:48033)
0812 01:15:07.058926 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:07.059013 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":241,"threads_started":1}
I20260812 01:15:07.102015 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:07.101305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:41475)
0812 01:15:07.101766 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:07.101849 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":235,"threads_started":1}
I20260812 01:15:07.585140 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:07.584348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:36887)
0812 01:15:07.584868 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:07.584960 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":265,"threads_started":1}
I20260812 01:15:08.061031 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:08.060263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:42289)
0812 01:15:08.060768 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:08.060853 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":274,"threads_started":1}
I20260812 01:15:08.103909 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:08.103022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:57113)
0812 01:15:08.103607 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:08.103727 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":295,"threads_started":1}
I20260812 01:15:08.586884 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:08.586081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:35351)
0812 01:15:08.586607 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:08.586698 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":250,"threads_started":1}
I20260812 01:15:09.062990 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:09.062082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:40901)
0812 01:15:09.062695 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:09.062795 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":269,"threads_started":1}
I20260812 01:15:09.105680 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:09.104903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:54261)
0812 01:15:09.105407 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:09.105498 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":271,"threads_started":1}
I20260812 01:15:09.588654 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:09.587852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:39707)
0812 01:15:09.588347 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:09.588430 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":276,"threads_started":1}
I20260812 01:15:10.064739 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:10.064005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:56733)
0812 01:15:10.064477 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:10.064562 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":254,"threads_started":1}
I20260812 01:15:10.107743 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:10.106771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:41755)
0812 01:15:10.107368 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:10.107480 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":298,"threads_started":1}
I20260812 01:15:10.590528 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:10.589722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:53037)
0812 01:15:10.590228 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:10.590333 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":256,"threads_started":1}
I20260812 01:15:11.066649 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:11.065772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:33687)
0812 01:15:11.066323 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:11.066422 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":253,"threads_started":1}
I20260812 01:15:11.109437 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:11.108677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:59291)
0812 01:15:11.109177 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:11.109270 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":240,"threads_started":1}
I20260812 01:15:11.592504 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:11.591625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:57885)
0812 01:15:11.592193 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:11.592295 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":268,"threads_started":1}
I20260812 01:15:12.068504 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:12.067763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:37233)
0812 01:15:12.068249 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:12.068337 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":257,"threads_started":1}
I20260812 01:15:12.111471 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:12.110484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:51341)
0812 01:15:12.111042 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:12.111185 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":278,"threads_started":1}
I20260812 01:15:12.594408 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:12.593511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:51221)
0812 01:15:12.594100 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:12.594206 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":304,"threads_started":1}
I20260812 01:15:13.070673 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:13.069749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:36687)
0812 01:15:13.070348 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:13.070469 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":337,"threads_started":1}
I20260812 01:15:13.113291 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:13.112463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:56167)
0812 01:15:13.112992 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:13.113099 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":251,"threads_started":1}
I20260812 01:15:13.596418 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:13.595543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:44487)
0812 01:15:13.596108 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:13.596212 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":289,"threads_started":1}
I20260812 01:15:14.072466 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:14.071719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:42723)
0812 01:15:14.072188 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:14.072275 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":226,"threads_started":1}
I20260812 01:15:14.115157 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:14.114395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.1:43567)
0812 01:15:14.114870 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:14.114961 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":264,"threads_started":1}
I20260812 01:15:14.598448 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:14.597686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.2:52709)
0812 01:15:14.598137 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:14.598250 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":221,"threads_started":1}
I20260812 01:15:15.049104 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 14876
I20260812 01:15:15.073997 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:15.050393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.20.151.1:45173)
0812 01:15:15.051119 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:15.051139 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:15.051189 (+ 50us) server_negotiation.cc:407] Connection header received
0812 01:15:15.053439 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:15.053450 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:15.053897 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:15.054196 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:15.055774 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:15.057079 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:15.058454 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:15.059165 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:15.059361 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:15.067009 (+ 7648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:15.067019 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:15.067039 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:15.067090 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:15.070095 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:15.070911 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:15.070917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:15.070925 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:15.071114 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:15.071580 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:15.071584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:15.071586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:15.072228 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:15.072484 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:15.073228 (+ 744us) server_negotiation.cc:299] Negotiation successful
0812 01:15:15.073682 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":370,"threads_started":1}
I20260812 01:15:15.074460 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:15.073699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:46709 (local address 127.20.151.3:53657)
0812 01:15:15.074195 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:15.074283 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":246,"threads_started":1}
I20260812 01:15:15.075534 15134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:15:15.161274 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 14876
I20260812 01:15:15.188578 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 15143
I20260812 01:15:15.207242 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:15.189560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.20.151.2:36217)
0812 01:15:15.190191 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:15.190206 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:15.190252 (+ 46us) server_negotiation.cc:407] Connection header received
0812 01:15:15.191961 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:15.191974 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:15.192441 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:15.192758 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:15.194223 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:15.195699 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:15.197124 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:15.197873 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:15.198023 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:15.200816 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:15.200825 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:15.200842 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:15.200888 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:15.204063 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:15.204775 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:15.204782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:15.204790 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:15.204978 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:15.205335 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:15.205340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:15.205343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:15.205940 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:15.206196 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:15.206550 (+ 354us) server_negotiation.cc:299] Negotiation successful
0812 01:15:15.206936 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":324,"threads_started":1}
I20260812 01:15:15.208648 15350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:15:15.294631 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 15143
I20260812 01:15:15.320282 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 15363
I20260812 01:15:15.339174 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:15.321289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.20.151.3:33415)
0812 01:15:15.321952 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:15.321966 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:15.322011 (+ 45us) server_negotiation.cc:407] Connection header received
0812 01:15:15.323749 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:15.323760 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:15.324217 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:15.324562 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:15.326115 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:15.327523 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:15.328804 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:15.329510 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:15.329700 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:15.332474 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:15.332483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:15.332500 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:15.332544 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:15.335655 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:15.336424 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:15.336432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:15.336441 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:15.336642 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:15.337050 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:15.337054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:15.337056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:15.337623 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:15.337946 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:15.338385 (+ 439us) server_negotiation.cc:299] Negotiation successful
0812 01:15:15.338854 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":282,"threads_started":1}
I20260812 01:15:15.340653 15572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:15:15.428155 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 15363
2026-08-12T01:15:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (26226 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:15:15 dist-test-slave-21b9 krb5kdc[15666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:15:15 dist-test-slave-21b9 krb5kdc[15666](info): set up 2 sockets
Aug 12 01:15:15 dist-test-slave-21b9 krb5kdc[15666](info): commencing operation
krb5kdc: starting...
W20260812 01:15:16.733554 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.004s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:15:16 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:15:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:15:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:15:17.081004 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:36143
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:36143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:15:17.437269 15694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:15:17.437580 15694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:15:17.437734 15694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:15:17.445109 15694 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 01:15:17.445212 15694 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 01:15:17.445255 15694 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 01:15:17.445328 15694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:15:17.445399 15694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:15:17.445505 15694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:15:17.445562 15694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:15:17.445640 15694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:15:17.456132 15694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:36143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:36143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:15:17.457504 15694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:15:17.458858 15694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:15:17.470813 15706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:17.470814 15704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:17.470834 15703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:15:17.471020 15694 server_base.cc:1034] running on GCE node
I20260812 01:15:17.472196 15694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:15:17.473618 15694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:15:17.474858 15694 hybrid_clock.cc:648] HybridClock initialized: now 1786497317474779 us; error 50 us; skew 500 ppm
I20260812 01:15:17.475205 15694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:15:17 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497317, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:15:17.478513 15694 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:15:17.480059 15694 webserver.cc:460] Webserver started at http://127.20.151.62:35579/ using document root <none> and password file <none>
I20260812 01:15:17.480767 15694 fs_manager.cc:359] Metadata directory not provided
I20260812 01:15:17.480872 15694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:15:17.481185 15694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:15:17.484315 15694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "ffe0e4afbfc54ebcb69b902b82d637ca"
format_stamp: "Formatted at 2026-08-12 01:15:17 on dist-test-slave-21b9"
server_key: "f9a7ae0c2ce22fdaceeaed83ea66e8c0"
server_key_iv: "8468ba1a4198d9ae383942ca1a0b1e8f"
server_key_version: "encryptionkey@0"
I20260812 01:15:17.485098 15694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "ffe0e4afbfc54ebcb69b902b82d637ca"
format_stamp: "Formatted at 2026-08-12 01:15:17 on dist-test-slave-21b9"
server_key: "f9a7ae0c2ce22fdaceeaed83ea66e8c0"
server_key_iv: "8468ba1a4198d9ae383942ca1a0b1e8f"
server_key_version: "encryptionkey@0"
I20260812 01:15:17.490327 15694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:15:17.493949 15717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:17.495462 15694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:15:17.495635 15694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "ffe0e4afbfc54ebcb69b902b82d637ca"
format_stamp: "Formatted at 2026-08-12 01:15:17 on dist-test-slave-21b9"
server_key: "f9a7ae0c2ce22fdaceeaed83ea66e8c0"
server_key_iv: "8468ba1a4198d9ae383942ca1a0b1e8f"
server_key_version: "encryptionkey@0"
I20260812 01:15:17.495855 15694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:15:17.508800 15694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:15:17.512984 15694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:15:17.513253 15694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:15:17.532282 15694 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:36143
I20260812 01:15:17.532308 15795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:36143 every 8 connection(s)
I20260812 01:15:17.533705 15694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:15:17.535688 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 15694
I20260812 01:15:17.535938 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:15:17.536813 21084 external_mini_cluster.cc:1428] Setting key d38d842606c805f0e4c0c7a9c04cc2ea
I20260812 01:15:17.538393 15796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 01:15:17 dist-test-slave-21b9 krb5kdc[15666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:15:17.552879 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Bootstrap starting.
I20260812 01:15:17.556349 15796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Neither blocks nor log segments found. Creating new log.
I20260812 01:15:17.557412 15796 log.cc:824] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Log is configured to *not* fsync() on all Append() calls
I20260812 01:15:17.560343 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: No bootstrap required, opened a new log
I20260812 01:15:17.560925 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:17.538983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.20.151.62:36143)
0812 01:15:17.539791 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:17.539806 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:17.539857 (+ 51us) server_negotiation.cc:407] Connection header received
0812 01:15:17.541711 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:17.541734 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:17.542134 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:17.542460 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:17.544567 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:17.545964 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:17.547858 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:17.548685 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:17.548853 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:17.553728 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:17.553736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:17.553752 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:17.553797 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:17.556855 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:17.557566 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:17.557575 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:17.557583 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:17.557759 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:17.558145 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:17.558149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:17.558151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:17.558703 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:17.558982 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:17.559508 (+ 526us) server_negotiation.cc:299] Negotiation successful
0812 01:15:17.559908 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":573,"thread_start_us":360,"threads_started":1}
I20260812 01:15:17.568277 15796 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:15:17.568647 15796 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:15:17.568739 15796 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe0e4afbfc54ebcb69b902b82d637ca, State: Initialized, Role: FOLLOWER
I20260812 01:15:17.569278 15796 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:15:17.569522 15796 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:15:17.569645 15796 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:15:17.569798 15796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:15:17.572058 15796 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:15:17.572645 15796 leader_election.cc:302] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffe0e4afbfc54ebcb69b902b82d637ca; no voters:
I20260812 01:15:17.573284 15796 leader_election.cc:288] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:15:17.573372 15803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:15:17.574142 15803 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 1 LEADER]: Becoming Leader. State: Replica: ffe0e4afbfc54ebcb69b902b82d637ca, State: Running, Role: LEADER
I20260812 01:15:17.574656 15803 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:15:17.575402 15796 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:15:17.577332 15810 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffe0e4afbfc54ebcb69b902b82d637ca. Latest consensus state: current_term: 1 leader_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } } }
I20260812 01:15:17.577252 15809 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } } }
I20260812 01:15:17.577704 15809 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: This master's current role is: LEADER
I20260812 01:15:17.580814 15810 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: This master's current role is: LEADER
I20260812 01:15:17.582368 15816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:15:17.587316 15816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:15:17.595903 15816 catalog_manager.cc:1347] Generated new cluster ID: 4080295ec7704313968f23d26eb83683
I20260812 01:15:17.596021 15816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:15:17.606837 15816 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:15:17.608515 15816 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:15:17.619066 15816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Generated new TSK 0
I20260812 01:15:17.620083 15816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:15:17.703497 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36143
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:15:18.063930 15838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:15:18.064235 15838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:15:18.064371 15838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:15:18.072160 15838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:15:18.072291 15838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:15:18.072450 15838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:15:18.083067 15838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:15:18.084447 15838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:15:18.085832 15838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:15:18.099589 15848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:18.099728 15851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:18.099592 15849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:15:18.100517 15838 server_base.cc:1034] running on GCE node
I20260812 01:15:18.101141 15838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:15:18.102133 15838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:15:18.103384 15838 hybrid_clock.cc:648] HybridClock initialized: now 1786497318103325 us; error 56 us; skew 500 ppm
I20260812 01:15:18.103724 15838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:15:18 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497318, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:15:18.106972 15838 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:15:18.108522 15838 webserver.cc:460] Webserver started at http://127.20.151.1:41557/ using document root <none> and password file <none>
I20260812 01:15:18.109239 15838 fs_manager.cc:359] Metadata directory not provided
I20260812 01:15:18.109365 15838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:15:18.109669 15838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:15:18.112690 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "758d529f16b8441f9d862de9f7138ca0"
format_stamp: "Formatted at 2026-08-12 01:15:18 on dist-test-slave-21b9"
server_key: "9575f37e102b2c18bd6aef86a7ef812b"
server_key_iv: "33aab32f960a8267f870d9f7f7811060"
server_key_version: "encryptionkey@0"
I20260812 01:15:18.113461 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "758d529f16b8441f9d862de9f7138ca0"
format_stamp: "Formatted at 2026-08-12 01:15:18 on dist-test-slave-21b9"
server_key: "9575f37e102b2c18bd6aef86a7ef812b"
server_key_iv: "33aab32f960a8267f870d9f7f7811060"
server_key_version: "encryptionkey@0"
I20260812 01:15:18.118804 15838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:15:18.122450 15866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:18.123867 15838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 01:15:18.124032 15838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "758d529f16b8441f9d862de9f7138ca0"
format_stamp: "Formatted at 2026-08-12 01:15:18 on dist-test-slave-21b9"
server_key: "9575f37e102b2c18bd6aef86a7ef812b"
server_key_iv: "33aab32f960a8267f870d9f7f7811060"
server_key_version: "encryptionkey@0"
I20260812 01:15:18.124234 15838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:15:18.147122 15838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:15:18.151647 15838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:15:18.151939 15838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:15:18.153019 15838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:15:18.154445 15838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:15:18.154526 15838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:18.154654 15838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:15:18.154711 15838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:18.185707 15838 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:41059
I20260812 01:15:18.185807 16047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:41059 every 8 connection(s)
I20260812 01:15:18.186991 15838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:15:18.189376 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 15838
I20260812 01:15:18.189599 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:15:18.190474 21084 external_mini_cluster.cc:1428] Setting key bf5fd9543a0106329740c5ac8dc5ab01
Aug 12 01:15:18 dist-test-slave-21b9 krb5kdc[15666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497318, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:15:18.210291 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:18.189777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:53745 (local address 127.20.151.62:36143)
0812 01:15:18.190309 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:18.190316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:18.191394 (+ 1078us) server_negotiation.cc:407] Connection header received
0812 01:15:18.193892 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:18.193896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:18.193970 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:18.194064 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:18.196185 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:18.196942 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:18.199124 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:18.199646 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:18.199746 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:18.204143 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:18.204149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:18.204153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:18.204188 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:18.206779 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:18.207745 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:18.207750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:18.207753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:18.207930 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:18.208400 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:18.208405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:18.208445 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:18.208793 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:18.208939 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:18.209690 (+ 751us) server_negotiation.cc:299] Negotiation successful
0812 01:15:18.209927 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":261,"threads_started":1}
I20260812 01:15:18.211117 16051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:18.190279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:53745)
0812 01:15:18.191189 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:18.191241 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:18.193563 (+ 2322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:18.194335 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:18.194347 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:18.195113 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:15:18.195983 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:18.196009 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:18.197125 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:18.197131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:18.198979 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:18.198996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:18.199892 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:18.199901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:18.200197 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:18.200988 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:15:18.201022 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:15:18.203863 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:15:18.206933 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:18.206942 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:18.206959 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:18.207608 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:18.208077 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:18.208081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:18.208083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:18.208286 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:18.208899 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:15:18.208905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:15:18.209254 (+ 349us) client_negotiation.cc:769] Sending connection context
0812 01:15:18.209485 (+ 231us) client_negotiation.cc:240] Negotiation successful
0812 01:15:18.209939 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":340,"threads_started":1}
I20260812 01:15:18.212749 16048 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36143
I20260812 01:15:18.213106 16048 heartbeater.cc:464] Registering TS with master...
I20260812 01:15:18.213973 16048 heartbeater.cc:511] Master 127.20.151.62:36143 requested a full tablet report, sending...
I20260812 01:15:18.216374 15745 ts_manager.cc:194] Registered new tserver with Master: 758d529f16b8441f9d862de9f7138ca0 (127.20.151.1:41059)
I20260812 01:15:18.217991 15745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:53745
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:15:18.266500 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36143
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:15:18.620613 16054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:15:18.620896 16054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:15:18.621001 16054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:15:18.628669 16054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:15:18.628785 16054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:15:18.628907 16054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:15:18.638747 16054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:15:18.640084 16054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:15:18.641371 16054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:15:18.654400 16059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:18.654372 16066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:18.654321 16061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:15:18.655186 16054 server_base.cc:1034] running on GCE node
I20260812 01:15:18.655823 16054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:15:18.656780 16054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:15:18.658003 16054 hybrid_clock.cc:648] HybridClock initialized: now 1786497318657947 us; error 45 us; skew 500 ppm
I20260812 01:15:18.658315 16054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:15:18 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497318, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:15:18.661413 16054 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:15:18.662904 16054 webserver.cc:460] Webserver started at http://127.20.151.2:41589/ using document root <none> and password file <none>
I20260812 01:15:18.663640 16054 fs_manager.cc:359] Metadata directory not provided
I20260812 01:15:18.663763 16054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:15:18.664060 16054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:15:18.666972 16054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "eea84dad4cde44079f02ff13631874ff"
format_stamp: "Formatted at 2026-08-12 01:15:18 on dist-test-slave-21b9"
server_key: "5c55df03227ba994b5c061f933caa1a0"
server_key_iv: "c5ab1135cd30d1a255e3413819e34b2c"
server_key_version: "encryptionkey@0"
I20260812 01:15:18.667770 16054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "eea84dad4cde44079f02ff13631874ff"
format_stamp: "Formatted at 2026-08-12 01:15:18 on dist-test-slave-21b9"
server_key: "5c55df03227ba994b5c061f933caa1a0"
server_key_iv: "c5ab1135cd30d1a255e3413819e34b2c"
server_key_version: "encryptionkey@0"
I20260812 01:15:18.672811 16054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:15:18.676288 16077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:18.677641 16054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:15:18.677800 16054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "eea84dad4cde44079f02ff13631874ff"
format_stamp: "Formatted at 2026-08-12 01:15:18 on dist-test-slave-21b9"
server_key: "5c55df03227ba994b5c061f933caa1a0"
server_key_iv: "c5ab1135cd30d1a255e3413819e34b2c"
server_key_version: "encryptionkey@0"
I20260812 01:15:18.677983 16054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:15:18.699353 16054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:15:18.703598 16054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:15:18.703873 16054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:15:18.704900 16054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:15:18.706344 16054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:15:18.706434 16054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:18.706570 16054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:15:18.706626 16054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:18.740315 16054 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:36949
I20260812 01:15:18.740375 16252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:36949 every 8 connection(s)
I20260812 01:15:18.742198 16054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:15:18.750272 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 16054
I20260812 01:15:18.750511 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:15:18.751075 21084 external_mini_cluster.cc:1428] Setting key 767ff529085183be9fea4bd319e08b8a
Aug 12 01:15:18 dist-test-slave-21b9 krb5kdc[15666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497318, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:15:18.765695 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:18.747534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:57193 (local address 127.20.151.62:36143)
0812 01:15:18.750628 (+ 3094us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:18.750634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:18.750655 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:15:18.750724 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:18.750730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:18.750808 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:18.750918 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:18.752755 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:18.753754 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:18.755365 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:18.755988 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:18.756115 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:18.759649 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:18.759655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:18.759659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:18.759698 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:18.762604 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:18.763391 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:18.763396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:18.763399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:18.763562 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:18.763985 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:18.763989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:18.763992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:18.764316 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:18.764446 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:18.765122 (+ 676us) server_negotiation.cc:299] Negotiation successful
0812 01:15:18.765339 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2971,"thread_start_us":2851,"threads_started":1}
I20260812 01:15:18.766543 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:18.747179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:57193)
0812 01:15:18.748042 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:18.748109 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:18.749934 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:18.751164 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:18.751175 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:18.751847 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:15:18.752579 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:18.752600 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:18.753925 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:18.753929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:18.755192 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:18.755205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:18.756197 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:18.756202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:18.756408 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:18.757040 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:15:18.757080 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:15:18.759401 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:15:18.762751 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:18.762757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:18.762771 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:18.763230 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:18.763690 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:18.763693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:18.763695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:18.763883 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:18.764422 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:15:18.764428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:15:18.764756 (+ 328us) client_negotiation.cc:769] Sending connection context
0812 01:15:18.764984 (+ 228us) client_negotiation.cc:240] Negotiation successful
0812 01:15:18.765393 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":313,"threads_started":1}
I20260812 01:15:18.768128 16253 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36143
I20260812 01:15:18.768467 16253 heartbeater.cc:464] Registering TS with master...
I20260812 01:15:18.769218 16253 heartbeater.cc:511] Master 127.20.151.62:36143 requested a full tablet report, sending...
I20260812 01:15:18.770732 15745 ts_manager.cc:194] Registered new tserver with Master: eea84dad4cde44079f02ff13631874ff (127.20.151.2:36949)
I20260812 01:15:18.771802 15745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:57193
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:15:18.825923 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36143
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:15:19.173000 16265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:15:19.173303 16265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:15:19.173446 16265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:15:19.180897 16265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:15:19.181030 16265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:15:19.181181 16265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:15:19.191409 16265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:15:19.192700 16265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:15:19.194005 16265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:15:19.207455 16273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:19.207624 16272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:19.207625 16279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:15:19.208825 16265 server_base.cc:1034] running on GCE node
I20260812 01:15:19.209491 16265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:15:19.210512 16265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:15:19.211727 16265 hybrid_clock.cc:648] HybridClock initialized: now 1786497319211669 us; error 51 us; skew 500 ppm
I20260812 01:15:19.212044 16265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:15:19 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497319, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:15:19.215449 16265 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:15:19.216956 16265 webserver.cc:460] Webserver started at http://127.20.151.3:37107/ using document root <none> and password file <none>
I20260812 01:15:19.217692 16265 fs_manager.cc:359] Metadata directory not provided
I20260812 01:15:19.217828 16265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:15:19.218142 16265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:15:19.221868 16048 heartbeater.cc:503] Master 127.20.151.62:36143 was elected leader, sending a full tablet report...
I20260812 01:15:19.221679 16265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "6c48c0cf90d0486aba167ad4b036c84e"
format_stamp: "Formatted at 2026-08-12 01:15:19 on dist-test-slave-21b9"
server_key: "77fa057286db48314f5ec2c63e9795e7"
server_key_iv: "c1d6a54ad6fbefd6c67b6dae53650855"
server_key_version: "encryptionkey@0"
I20260812 01:15:19.222550 16265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "6c48c0cf90d0486aba167ad4b036c84e"
format_stamp: "Formatted at 2026-08-12 01:15:19 on dist-test-slave-21b9"
server_key: "77fa057286db48314f5ec2c63e9795e7"
server_key_iv: "c1d6a54ad6fbefd6c67b6dae53650855"
server_key_version: "encryptionkey@0"
I20260812 01:15:19.228534 16265 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 01:15:19.232285 16289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:19.233814 16265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:15:19.233987 16265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "6c48c0cf90d0486aba167ad4b036c84e"
format_stamp: "Formatted at 2026-08-12 01:15:19 on dist-test-slave-21b9"
server_key: "77fa057286db48314f5ec2c63e9795e7"
server_key_iv: "c1d6a54ad6fbefd6c67b6dae53650855"
server_key_version: "encryptionkey@0"
I20260812 01:15:19.234203 16265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:15:19.248912 16265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:15:19.253505 16265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:15:19.253798 16265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:15:19.254801 16265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:15:19.256171 16265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:15:19.256247 16265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:19.256362 16265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:15:19.256417 16265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:19.286974 16265 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:35213
I20260812 01:15:19.287019 16469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:35213 every 8 connection(s)
I20260812 01:15:19.288267 16265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:15:19.288784 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 16265
I20260812 01:15:19.289016 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:15:19.289620 21084 external_mini_cluster.cc:1428] Setting key 5dd02f58acf1621b6574e8ec14bdbfcd
Aug 12 01:15:19 dist-test-slave-21b9 krb5kdc[15666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497319, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:15:19.309168 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.291174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:48027 (local address 127.20.151.62:36143)
0812 01:15:19.291683 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:19.291689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:19.292911 (+ 1222us) server_negotiation.cc:407] Connection header received
0812 01:15:19.294763 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:19.294767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:19.294835 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:19.294921 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:19.296583 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.297384 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.298981 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.299441 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.299537 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.303115 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:19.303120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:19.303124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:19.303159 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:19.306253 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:19.306980 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:19.306984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:19.306986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:19.307118 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:19.307556 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:19.307559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:19.307561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:19.307855 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:19.307971 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:19.308709 (+ 738us) server_negotiation.cc:299] Negotiation successful
0812 01:15:19.308880 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":241,"threads_started":1}
I20260812 01:15:19.310153 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.291864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:48027)
0812 01:15:19.292736 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:19.292782 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:19.294518 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:19.295154 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:19.295163 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:19.295817 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:15:19.296430 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.296449 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.297614 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.297618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.298859 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.298871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.299690 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.299696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.299906 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.300542 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:15:19.300565 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:15:19.302890 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:15:19.306378 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:19.306386 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:19.306401 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:19.306887 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:19.307196 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:19.307199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:19.307201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:19.307461 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:19.307979 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:15:19.307986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:15:19.308344 (+ 358us) client_negotiation.cc:769] Sending connection context
0812 01:15:19.308623 (+ 279us) client_negotiation.cc:240] Negotiation successful
0812 01:15:19.309027 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":323,"threads_started":1}
I20260812 01:15:19.311610 16470 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36143
I20260812 01:15:19.311896 16470 heartbeater.cc:464] Registering TS with master...
I20260812 01:15:19.312651 16470 heartbeater.cc:511] Master 127.20.151.62:36143 requested a full tablet report, sending...
I20260812 01:15:19.314062 15745 ts_manager.cc:194] Registered new tserver with Master: 6c48c0cf90d0486aba167ad4b036c84e (127.20.151.3:35213)
I20260812 01:15:19.314885 15745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:48027
I20260812 01:15:19.318254 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:15:19.336079 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.322980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.20.151.62:36143)
0812 01:15:19.323142 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:19.323149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:19.323610 (+ 461us) server_negotiation.cc:407] Connection header received
0812 01:15:19.323785 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:19.323791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:19.323861 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:19.323955 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:19.325451 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.326203 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.327761 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.328189 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.328302 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.330859 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:19.330864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:19.330868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:19.330903 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:19.333321 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:19.334036 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:19.334040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:19.334043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:19.334179 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:19.334574 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:19.334579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:19.334582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:19.334906 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:19.335153 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:19.335533 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 01:15:19.335765 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 01:15:19.339994 15745 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:15:19.346859 15745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:15:19.370877 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.359314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:36949 (local address 127.0.0.1:49552)
0812 01:15:19.360185 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:19.360232 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:19.360400 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:19.362329 (+ 1929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:19.362333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:19.362377 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:15:19.362886 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.362901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.365300 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.365306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.367833 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.367845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.369726 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.369733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.369938 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.369997 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 01:15:19.370159 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 01:15:19.370363 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":135,"spinlock_wait_cycles":4608,"thread_start_us":978,"threads_started":1}
I20260812 01:15:19.371006 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.360166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.20.151.1:41059)
0812 01:15:19.361667 (+ 1501us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:19.361676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:19.361708 (+ 32us) server_negotiation.cc:407] Connection header received
0812 01:15:19.361808 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:19.361816 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:19.362076 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:19.363437 (+ 1361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:15:19.363653 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.365160 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.368390 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.369483 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.369674 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.369909 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:19.370485 (+ 576us) server_negotiation.cc:299] Negotiation successful
0812 01:15:19.370675 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1291,"thread_start_us":2279,"threads_started":1}
I20260812 01:15:19.370968 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.361557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:35213 (local address 127.0.0.1:49616)
0812 01:15:19.362118 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:19.362135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:19.362245 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:19.363648 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:19.363653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:19.363677 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:15:19.364030 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.364039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.365584 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.365588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.368209 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.368224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.370113 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.370119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.370322 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.370352 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 01:15:19.370437 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 01:15:19.370570 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":118,"thread_start_us":764,"threads_started":1}
I20260812 01:15:19.372018 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.359527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41059 (local address 127.0.0.1:46220)
0812 01:15:19.360974 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:19.360997 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:19.361102 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:19.362317 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:19.362324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:19.362380 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:15:19.363232 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.363248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.365335 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.365339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.368218 (+ 2879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.368228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.370072 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.370080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.370264 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.370323 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 01:15:19.370401 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 01:15:19.371655 (+ 1254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"mutex_wait_us":141,"spinlock_wait_cycles":9600,"thread_start_us":1199,"threads_started":1}
I20260812 01:15:19.372051 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.362143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49616 (local address 127.20.151.3:35213)
0812 01:15:19.363080 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:19.363094 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:19.363119 (+ 25us) server_negotiation.cc:407] Connection header received
0812 01:15:19.363211 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:19.363217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:19.363442 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:19.364198 (+ 756us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:15:19.364393 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.365427 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.368376 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.369889 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.370020 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.370231 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:19.371101 (+ 870us) server_negotiation.cc:299] Negotiation successful
0812 01:15:19.371262 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":644,"threads_started":1}
I20260812 01:15:19.372247 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.359532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.20.151.2:36949)
0812 01:15:19.361804 (+ 2272us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:19.361811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:19.361832 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:15:19.361910 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:19.361916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:19.362095 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:19.363285 (+ 1190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:15:19.363554 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.365161 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.367999 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.369500 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.369658 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.369871 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:19.371741 (+ 1870us) server_negotiation.cc:299] Negotiation successful
0812 01:15:19.371918 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2116,"thread_start_us":1998,"threads_started":1}
I20260812 01:15:19.375550 16146 tablet_service.cc:1467] Processing CreateTablet for tablet dd90deb949c3432298fc1445774bef6b (DEFAULT_TABLE table=test-table [id=aa53b615106f47a99a0fb275a3a272c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:15:19.375748 15948 tablet_service.cc:1467] Processing CreateTablet for tablet dd90deb949c3432298fc1445774bef6b (DEFAULT_TABLE table=test-table [id=aa53b615106f47a99a0fb275a3a272c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:15:19.375748 16370 tablet_service.cc:1467] Processing CreateTablet for tablet dd90deb949c3432298fc1445774bef6b (DEFAULT_TABLE table=test-table [id=aa53b615106f47a99a0fb275a3a272c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:15:19.377056 16146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd90deb949c3432298fc1445774bef6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:15:19.377122 15948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd90deb949c3432298fc1445774bef6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:15:19.377117 16370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd90deb949c3432298fc1445774bef6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:15:19.390924 16501 tablet_bootstrap.cc:492] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: Bootstrap starting.
I20260812 01:15:19.394399 16501 tablet_bootstrap.cc:654] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: Neither blocks nor log segments found. Creating new log.
I20260812 01:15:19.395576 16501 log.cc:824] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: Log is configured to *not* fsync() on all Append() calls
I20260812 01:15:19.396060 16502 tablet_bootstrap.cc:492] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: Bootstrap starting.
I20260812 01:15:19.397742 16503 tablet_bootstrap.cc:492] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: Bootstrap starting.
I20260812 01:15:19.399889 16502 tablet_bootstrap.cc:654] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: Neither blocks nor log segments found. Creating new log.
I20260812 01:15:19.401217 16502 log.cc:824] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: Log is configured to *not* fsync() on all Append() calls
I20260812 01:15:19.401259 16503 tablet_bootstrap.cc:654] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: Neither blocks nor log segments found. Creating new log.
I20260812 01:15:19.402402 16501 tablet_bootstrap.cc:492] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: No bootstrap required, opened a new log
I20260812 01:15:19.402415 16503 log.cc:824] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: Log is configured to *not* fsync() on all Append() calls
I20260812 01:15:19.402729 16501 ts_tablet_manager.cc:1397] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260812 01:15:19.405237 16502 tablet_bootstrap.cc:492] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: No bootstrap required, opened a new log
I20260812 01:15:19.405511 16502 ts_tablet_manager.cc:1397] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260812 01:15:19.405507 16503 tablet_bootstrap.cc:492] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: No bootstrap required, opened a new log
I20260812 01:15:19.405766 16503 ts_tablet_manager.cc:1397] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 01:15:19.413167 16502 raft_consensus.cc:348] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:19.413264 16501 raft_consensus.cc:348] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:19.413570 16502 raft_consensus.cc:374] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:15:19.413666 16502 raft_consensus.cc:729] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758d529f16b8441f9d862de9f7138ca0, State: Initialized, Role: FOLLOWER
I20260812 01:15:19.413740 16501 raft_consensus.cc:374] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:15:19.413851 16501 raft_consensus.cc:729] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea84dad4cde44079f02ff13631874ff, State: Initialized, Role: FOLLOWER
I20260812 01:15:19.414290 16502 consensus_queue.cc:260] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:19.414554 16501 consensus_queue.cc:260] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:19.416189 16502 ts_tablet_manager.cc:1428] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20260812 01:15:19.416272 16503 raft_consensus.cc:348] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:19.416850 16503 raft_consensus.cc:374] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:15:19.416996 16503 raft_consensus.cc:729] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c48c0cf90d0486aba167ad4b036c84e, State: Initialized, Role: FOLLOWER
I20260812 01:15:19.417680 16503 consensus_queue.cc:260] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:19.417989 16501 ts_tablet_manager.cc:1428] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260812 01:15:19.418573 16253 heartbeater.cc:503] Master 127.20.151.62:36143 was elected leader, sending a full tablet report...
I20260812 01:15:19.419126 16503 ts_tablet_manager.cc:1428] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260812 01:15:19.419543 16470 heartbeater.cc:503] Master 127.20.151.62:36143 was elected leader, sending a full tablet report...
W20260812 01:15:19.439599 16049 tablet.cc:2369] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:15:19.495937 16255 tablet.cc:2369] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:15:19.540287 16472 tablet.cc:2369] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:15:19.639437 16509 raft_consensus.cc:482] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:15:19.639768 16509 raft_consensus.cc:504] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:19.641530 16509 leader_election.cc:288] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eea84dad4cde44079f02ff13631874ff (127.20.151.2:36949), 6c48c0cf90d0486aba167ad4b036c84e (127.20.151.3:35213)
I20260812 01:15:19.648272 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.642057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:49839 (local address 127.20.151.3:35213)
0812 01:15:19.642277 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:19.642283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:19.643034 (+ 751us) server_negotiation.cc:407] Connection header received
0812 01:15:19.643232 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:19.643236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:19.643323 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:19.643399 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:15:19.644027 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.644763 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.646507 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.647320 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.647478 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.647553 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:19.647904 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 01:15:19.648032 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 01:15:19.648288 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.641996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:41405 (local address 127.20.151.2:36949)
0812 01:15:19.642233 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:19.642240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:19.642966 (+ 726us) server_negotiation.cc:407] Connection header received
0812 01:15:19.643167 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:19.643171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:19.643234 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:19.643331 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:15:19.643885 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.644588 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.646441 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.647260 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.647435 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.647514 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:19.647890 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 01:15:19.648021 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 01:15:19.648504 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.641777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:36949 (local address 127.20.151.1:41405)
0812 01:15:19.642844 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:19.642860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:19.642995 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:19.643350 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:19.643355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:19.643379 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:15:19.643736 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.643751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.644722 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.644725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.646282 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.646296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.647527 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.647533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.647737 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.647762 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:15:19.647845 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 01:15:19.647983 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":55,"thread_start_us":676,"threads_started":1}
I20260812 01:15:19.648479 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.642021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:35213 (local address 127.20.151.1:49839)
0812 01:15:19.642941 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:19.642959 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:19.643099 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:19.643445 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:19.643451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:19.643490 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:15:19.643874 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.643884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.644904 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.644908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.646352 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:19.646365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.647566 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.647573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:19.647752 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.647777 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:15:19.647856 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 01:15:19.647984 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":29,"thread_start_us":498,"threads_started":1}
I20260812 01:15:19.649654 16399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90deb949c3432298fc1445774bef6b" candidate_uuid: "758d529f16b8441f9d862de9f7138ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c48c0cf90d0486aba167ad4b036c84e" is_pre_election: true
I20260812 01:15:19.649667 16181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90deb949c3432298fc1445774bef6b" candidate_uuid: "758d529f16b8441f9d862de9f7138ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea84dad4cde44079f02ff13631874ff" is_pre_election: true
I20260812 01:15:19.650151 16181 raft_consensus.cc:2454] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758d529f16b8441f9d862de9f7138ca0 in term 0.
I20260812 01:15:19.650159 16399 raft_consensus.cc:2454] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758d529f16b8441f9d862de9f7138ca0 in term 0.
I20260812 01:15:19.650858 15874 leader_election.cc:302] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758d529f16b8441f9d862de9f7138ca0, eea84dad4cde44079f02ff13631874ff; no voters:
I20260812 01:15:19.651302 16509 raft_consensus.cc:2781] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:15:19.651476 16509 raft_consensus.cc:482] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:15:19.651590 16509 raft_consensus.cc:3037] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:15:19.653879 16509 raft_consensus.cc:504] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:19.654524 16509 leader_election.cc:288] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [CANDIDATE]: Term 1 election: Requested vote from peers eea84dad4cde44079f02ff13631874ff (127.20.151.2:36949), 6c48c0cf90d0486aba167ad4b036c84e (127.20.151.3:35213)
I20260812 01:15:19.654933 16181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90deb949c3432298fc1445774bef6b" candidate_uuid: "758d529f16b8441f9d862de9f7138ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea84dad4cde44079f02ff13631874ff"
I20260812 01:15:19.654979 16399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90deb949c3432298fc1445774bef6b" candidate_uuid: "758d529f16b8441f9d862de9f7138ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c48c0cf90d0486aba167ad4b036c84e"
I20260812 01:15:19.655148 16181 raft_consensus.cc:3037] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:15:19.655179 16399 raft_consensus.cc:3037] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:15:19.657330 16181 raft_consensus.cc:2454] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758d529f16b8441f9d862de9f7138ca0 in term 1.
I20260812 01:15:19.657332 16399 raft_consensus.cc:2454] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758d529f16b8441f9d862de9f7138ca0 in term 1.
I20260812 01:15:19.657702 15874 leader_election.cc:302] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758d529f16b8441f9d862de9f7138ca0, eea84dad4cde44079f02ff13631874ff; no voters:
I20260812 01:15:19.658008 16509 raft_consensus.cc:2781] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:15:19.658432 16509 raft_consensus.cc:686] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 1 LEADER]: Becoming Leader. State: Replica: 758d529f16b8441f9d862de9f7138ca0, State: Running, Role: LEADER
I20260812 01:15:19.658896 16509 consensus_queue.cc:237] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:19.663738 15743 catalog_manager.cc:5495] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 758d529f16b8441f9d862de9f7138ca0 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "758d529f16b8441f9d862de9f7138ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } health_report { overall_health: UNKNOWN } } }
I20260812 01:15:19.737064 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.731891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.20.151.1:41059)
0812 01:15:19.732116 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:19.732122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:19.732167 (+ 45us) server_negotiation.cc:407] Connection header received
0812 01:15:19.732350 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:19.732354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:19.732433 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:19.732555 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:15:19.732872 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.733628 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.734958 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.735423 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.735515 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.736004 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:15:19.736528 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:15:19.736682 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:19.736721 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 01:15:19.736790 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:15:19.745957 16181 raft_consensus.cc:1261] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff [term 1 FOLLOWER]: Refusing update from remote peer 758d529f16b8441f9d862de9f7138ca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:15:19.746074 16399 raft_consensus.cc:1261] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e [term 1 FOLLOWER]: Refusing update from remote peer 758d529f16b8441f9d862de9f7138ca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:15:19.746975 16509 consensus_queue.cc:1035] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:15:19.747635 16509 consensus_queue.cc:1035] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:15:19.776443 15743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:15:19.776787 15743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:15:19.780782 15743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Sending DeleteTablet for 3 replicas of tablet dd90deb949c3432298fc1445774bef6b
I20260812 01:15:19.782048 15948 tablet_service.cc:1514] Processing DeleteTablet for tablet dd90deb949c3432298fc1445774bef6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:15:19 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:46220
I20260812 01:15:19.782027 16146 tablet_service.cc:1514] Processing DeleteTablet for tablet dd90deb949c3432298fc1445774bef6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:15:19 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:49552
I20260812 01:15:19.782087 16370 tablet_service.cc:1514] Processing DeleteTablet for tablet dd90deb949c3432298fc1445774bef6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:15:19 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:49616
I20260812 01:15:19.782373 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 15838
I20260812 01:15:19.800348 16541 tablet_replica.cc:330] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: stopping tablet replica
I20260812 01:15:19.801029 16541 raft_consensus.cc:2229] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:15:19.801703 16539 tablet_replica.cc:330] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: stopping tablet replica
I20260812 01:15:19.802135 16539 raft_consensus.cc:2229] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:15:19.802883 16539 raft_consensus.cc:2258] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:15:19.803738 16541 raft_consensus.cc:2258] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 12 01:15:19 dist-test-slave-21b9 krb5kdc[15666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:15:19.805768 16539 ts_tablet_manager.cc:1905] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:15:19.806465 16541 ts_tablet_manager.cc:1905] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:15:19.814356 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.783583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.20.151.1:41059)
0812 01:15:19.783792 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:19.783799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:19.793164 (+ 9365us) server_negotiation.cc:407] Connection header received
0812 01:15:19.793396 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:19.793401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:19.793483 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:19.793594 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:19.795598 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.796875 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.799046 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.799891 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.799997 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.806022 (+ 6025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:19.806038 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:19.806057 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:19.806120 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:19.810231 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:19.811310 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:19.811321 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:19.811324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:19.811529 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:19.812053 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:19.812071 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:19.812074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:19.812559 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:19.812878 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:19.813653 (+ 775us) server_negotiation.cc:299] Negotiation successful
0812 01:15:19.813944 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 01:15:19.814579 16539 ts_tablet_manager.cc:1918] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:15:19.814811 16539 log.cc:1196] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/dd90deb949c3432298fc1445774bef6b
I20260812 01:15:19.815435 16539 ts_tablet_manager.cc:1939] T dd90deb949c3432298fc1445774bef6b P 758d529f16b8441f9d862de9f7138ca0: Deleting consensus metadata
I20260812 01:15:19.815613 16041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:15:19.815780 16540 tablet_replica.cc:330] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: stopping tablet replica
I20260812 01:15:19.816295 16540 raft_consensus.cc:2229] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:15:19.817099 16540 raft_consensus.cc:2258] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:15:19.821103 16541 ts_tablet_manager.cc:1918] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:15:19.821295 16541 log.cc:1196] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/dd90deb949c3432298fc1445774bef6b
I20260812 01:15:19.821440 16540 ts_tablet_manager.cc:1905] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:15:19.821837 16541 ts_tablet_manager.cc:1939] T dd90deb949c3432298fc1445774bef6b P 6c48c0cf90d0486aba167ad4b036c84e: Deleting consensus metadata
I20260812 01:15:19.823853 15720 catalog_manager.cc:4841] TS 6c48c0cf90d0486aba167ad4b036c84e (127.20.151.3:35213): tablet dd90deb949c3432298fc1445774bef6b (table test-table [id=aa53b615106f47a99a0fb275a3a272c0]) successfully deleted
I20260812 01:15:19.827054 16540 ts_tablet_manager.cc:1918] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:15:19.827193 16540 log.cc:1196] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/dd90deb949c3432298fc1445774bef6b
I20260812 01:15:19.827630 16540 ts_tablet_manager.cc:1939] T dd90deb949c3432298fc1445774bef6b P eea84dad4cde44079f02ff13631874ff: Deleting consensus metadata
I20260812 01:15:19.829159 15721 catalog_manager.cc:4841] TS eea84dad4cde44079f02ff13631874ff (127.20.151.2:36949): tablet dd90deb949c3432298fc1445774bef6b (table test-table [id=aa53b615106f47a99a0fb275a3a272c0]) successfully deleted
I20260812 01:15:19.914525 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 15838
I20260812 01:15:19.914669 15720 catalog_manager.cc:4841] TS 758d529f16b8441f9d862de9f7138ca0 (127.20.151.1:41059): tablet dd90deb949c3432298fc1445774bef6b (table test-table [id=aa53b615106f47a99a0fb275a3a272c0]) successfully deleted
I20260812 01:15:19.938359 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 16054
Aug 12 01:15:19 dist-test-slave-21b9 krb5kdc[15666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:15:19.953771 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:19.939249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.20.151.2:36949)
0812 01:15:19.939443 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:19.939448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:19.939473 (+ 25us) server_negotiation.cc:407] Connection header received
0812 01:15:19.939622 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:19.939625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:19.939700 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:19.939783 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:19.941060 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.941846 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.943189 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:19.943681 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:19.943776 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:19.948080 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:19.948100 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:19.948117 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:19.948154 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:19.951032 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:19.951883 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:19.951891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:19.951893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:19.952036 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:19.952336 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:19.952339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:19.952340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:19.952741 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:19.952971 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:19.953178 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 01:15:19.953379 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:15:19.954443 16245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:15:20.049650 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 16054
I20260812 01:15:20.077658 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 16265
Aug 12 01:15:20 dist-test-slave-21b9 krb5kdc[15666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:15:20.097151 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:20.078452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.20.151.3:35213)
0812 01:15:20.078737 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:20.078743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:20.078765 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:15:20.078835 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:20.078841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:20.078913 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:20.079011 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:20.080398 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:20.081139 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:20.082495 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:20.082885 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:20.083007 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:20.091503 (+ 8496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:20.091523 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:20.091541 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:20.091584 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:20.094264 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:20.095003 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:20.095012 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:20.095015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:20.095169 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:20.095568 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:20.095571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:20.095573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:20.095950 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:20.096251 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:20.096601 (+ 350us) server_negotiation.cc:299] Negotiation successful
0812 01:15:20.096826 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:15:20.097855 16464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:15:20.189389 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 16265
I20260812 01:15:20.217548 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 15694
I20260812 01:15:20.218719 15790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:15:20.331540 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 15694
I20260812 01:15:40.361704 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:36143
--webserver_interface=127.20.151.62
--webserver_port=35579
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:36143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:15:40.725749 16552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:15:40.726096 16552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:15:40.726259 16552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:15:40.733990 16552 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 01:15:40.734138 16552 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 01:15:40.734216 16552 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 01:15:40.734301 16552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:15:40.734445 16552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:15:40.734570 16552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:15:40.734647 16552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:15:40.734735 16552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:15:40.745224 16552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:36143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:36143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=35579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:15:40.746624 16552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:15:40.748117 16552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:15:40.775048 16564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:40.775223 16560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:40.775089 16561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:15:40.776486 16552 server_base.cc:1034] running on GCE node
I20260812 01:15:40.777467 16552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:15:40.779166 16552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:15:40.780488 16552 hybrid_clock.cc:648] HybridClock initialized: now 1786497340780370 us; error 37 us; skew 500 ppm
I20260812 01:15:40.780975 16552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:15:40 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497340, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:15:40.784972 16552 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:15:40.786895 16552 webserver.cc:460] Webserver started at http://127.20.151.62:35579/ using document root <none> and password file <none>
I20260812 01:15:40.787837 16552 fs_manager.cc:359] Metadata directory not provided
I20260812 01:15:40.787986 16552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:15:40.794642 16552 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 01:15:40.797851 16571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:40.799350 16552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 01:15:40.799556 16552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "ffe0e4afbfc54ebcb69b902b82d637ca"
format_stamp: "Formatted at 2026-08-12 01:15:17 on dist-test-slave-21b9"
server_key: "f9a7ae0c2ce22fdaceeaed83ea66e8c0"
server_key_iv: "8468ba1a4198d9ae383942ca1a0b1e8f"
server_key_version: "encryptionkey@0"
I20260812 01:15:40.800689 16552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:15:40.821782 16552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:15:40.826480 16552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:15:40.826849 16552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:15:40.846073 16552 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:36143
I20260812 01:15:40.846151 16660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:36143 every 8 connection(s)
I20260812 01:15:40.847867 16552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:15:40.855487 16662 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:15:40.859001 16662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Bootstrap starting.
I20260812 01:15:40.860894 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 16552
I20260812 01:15:40.861730 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:41059
--local_ip_for_outbound_sockets=127.20.151.1
--tserver_master_addrs=127.20.151.62:36143
--webserver_port=41557
--webserver_interface=127.20.151.1
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:15:40.880369 16662 log.cc:824] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Log is configured to *not* fsync() on all Append() calls
I20260812 01:15:40.893407 16662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:15:40.894001 16662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Bootstrap complete.
I20260812 01:15:40.903870 16662 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:15:40.904852 16662 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe0e4afbfc54ebcb69b902b82d637ca, State: Initialized, Role: FOLLOWER
I20260812 01:15:40.905532 16662 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:15:40.905797 16662 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:15:40.905930 16662 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:15:40.906111 16662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 1 FOLLOWER]: Advancing to term 2
I20260812 01:15:40.910270 16662 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:15:40.910813 16662 leader_election.cc:302] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffe0e4afbfc54ebcb69b902b82d637ca; no voters:
I20260812 01:15:40.911474 16662 leader_election.cc:288] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 01:15:40.911616 16666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 2 FOLLOWER]: Leader election won for term 2
I20260812 01:15:40.912735 16666 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 2 LEADER]: Becoming Leader. State: Replica: ffe0e4afbfc54ebcb69b902b82d637ca, State: Running, Role: LEADER
I20260812 01:15:40.913313 16666 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:15:40.913818 16662 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:15:40.915951 16671 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffe0e4afbfc54ebcb69b902b82d637ca. Latest consensus state: current_term: 2 leader_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } } }
I20260812 01:15:40.916327 16671 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: This master's current role is: LEADER
I20260812 01:15:40.916942 16670 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } } }
I20260812 01:15:40.917250 16670 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: This master's current role is: LEADER
I20260812 01:15:40.928731 16675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:15:40.937614 16675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:15:40.939142 16675 catalog_manager.cc:1259] Loaded cluster ID: 4080295ec7704313968f23d26eb83683
I20260812 01:15:40.939333 16675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:15:40.943460 16675 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:15:40.944326 16699 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 01:15:40.945473 16675 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Loaded TSK: 0
I20260812 01:15:40.975687 16675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Generated new TSK 1
I20260812 01:15:40.976125 16675 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 01:15:41.240715 16664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:15:41.241019 16664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:15:41.241137 16664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:15:41.248600 16664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:15:41.248713 16664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:15:41.248854 16664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:15:41.258873 16664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:41059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=41557
--tserver_master_addrs=127.20.151.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:15:41.260253 16664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:15:41.261597 16664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:15:41.275856 16714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:41.276026 16718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:41.275878 16711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:15:41.277271 16664 server_base.cc:1034] running on GCE node
I20260812 01:15:41.277943 16664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:15:41.278996 16664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:15:41.280220 16664 hybrid_clock.cc:648] HybridClock initialized: now 1786497341280162 us; error 50 us; skew 500 ppm
I20260812 01:15:41.280558 16664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:15:41 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497341, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:15:41.287881 16664 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:15:41.289893 16664 webserver.cc:460] Webserver started at http://127.20.151.1:41557/ using document root <none> and password file <none>
I20260812 01:15:41.290647 16664 fs_manager.cc:359] Metadata directory not provided
I20260812 01:15:41.290787 16664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:15:41.296693 16664 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260812 01:15:41.299758 16729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:41.301193 16664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:15:41.301375 16664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "758d529f16b8441f9d862de9f7138ca0"
format_stamp: "Formatted at 2026-08-12 01:15:18 on dist-test-slave-21b9"
server_key: "9575f37e102b2c18bd6aef86a7ef812b"
server_key_iv: "33aab32f960a8267f870d9f7f7811060"
server_key_version: "encryptionkey@0"
I20260812 01:15:41.302384 16664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:15:41.326869 16664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:15:41.331470 16664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:15:41.331761 16664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:15:41.332795 16664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:15:41.334199 16664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:15:41.334307 16664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:41.334417 16664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:15:41.334497 16664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:41.366370 16664 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:41059
I20260812 01:15:41.366431 16905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:41059 every 8 connection(s)
I20260812 01:15:41.367957 16664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:15:41.377652 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 16664
I20260812 01:15:41.378505 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:36949
--local_ip_for_outbound_sockets=127.20.151.2
--tserver_master_addrs=127.20.151.62:36143
--webserver_port=41589
--webserver_interface=127.20.151.2
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:15:41 dist-test-slave-21b9 krb5kdc[15666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497341, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:15:41.395917 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:41.371101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:39219)
0812 01:15:41.375797 (+ 4696us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:41.375839 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:41.378076 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:41.379619 (+ 1543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:41.379631 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:41.380277 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:15:41.380815 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:41.380833 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:41.382497 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:41.382502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:41.383789 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:41.383803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:41.384937 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:41.384945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:41.385232 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:41.385907 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:15:41.385936 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:15:41.388286 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:15:41.391728 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:41.391737 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:41.391754 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:41.392202 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:41.392730 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:41.392734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:41.392737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:41.392936 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:41.393858 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:15:41.393864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:15:41.394191 (+ 327us) client_negotiation.cc:769] Sending connection context
0812 01:15:41.394421 (+ 230us) client_negotiation.cc:240] Negotiation successful
0812 01:15:41.394834 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4425,"thread_start_us":4215,"threads_started":1}
I20260812 01:15:41.396734 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:41.370976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:39219 (local address 127.20.151.62:36143)
0812 01:15:41.371666 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:41.371684 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:41.376003 (+ 4319us) server_negotiation.cc:407] Connection header received
0812 01:15:41.378672 (+ 2669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:41.378692 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:41.379059 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:41.379450 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:41.381046 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:41.382332 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:41.383935 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:41.384596 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:41.384911 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:41.388586 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:41.388595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:41.388611 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:41.388650 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:41.391584 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:41.392345 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:41.392355 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:41.392363 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:41.392596 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:41.393050 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:41.393055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:41.393058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:41.393733 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:41.393931 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:41.394683 (+ 752us) server_negotiation.cc:299] Negotiation successful
0812 01:15:41.395132 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":263,"threads_started":1}
I20260812 01:15:41.399559 16908 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36143
I20260812 01:15:41.399865 16908 heartbeater.cc:464] Registering TS with master...
I20260812 01:15:41.400604 16908 heartbeater.cc:511] Master 127.20.151.62:36143 requested a full tablet report, sending...
I20260812 01:15:41.402767 16597 ts_manager.cc:194] Registered new tserver with Master: 758d529f16b8441f9d862de9f7138ca0 (127.20.151.1:41059)
I20260812 01:15:41.404326 16597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:39219
W20260812 01:15:41.728957 16913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:15:41.729287 16913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:15:41.729434 16913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:15:41.736712 16913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:15:41.736835 16913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:15:41.736986 16913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:15:41.746944 16913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:36949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=41589
--tserver_master_addrs=127.20.151.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:15:41.748307 16913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:15:41.749657 16913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:15:41.762638 16922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:41.762701 16923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:41.762681 16925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:15:41.763921 16913 server_base.cc:1034] running on GCE node
I20260812 01:15:41.764539 16913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:15:41.765452 16913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:15:41.766672 16913 hybrid_clock.cc:648] HybridClock initialized: now 1786497341766621 us; error 32 us; skew 500 ppm
I20260812 01:15:41.767004 16913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:15:41 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497341, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:15:41.770375 16913 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:15:41.771901 16913 webserver.cc:460] Webserver started at http://127.20.151.2:41589/ using document root <none> and password file <none>
I20260812 01:15:41.772634 16913 fs_manager.cc:359] Metadata directory not provided
I20260812 01:15:41.772774 16913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:15:41.778366 16913 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 01:15:41.781273 16938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:41.782635 16913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:15:41.782799 16913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "eea84dad4cde44079f02ff13631874ff"
format_stamp: "Formatted at 2026-08-12 01:15:18 on dist-test-slave-21b9"
server_key: "5c55df03227ba994b5c061f933caa1a0"
server_key_iv: "c5ab1135cd30d1a255e3413819e34b2c"
server_key_version: "encryptionkey@0"
I20260812 01:15:41.783891 16913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:15:41.796571 16913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:15:41.801083 16913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:15:41.801365 16913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:15:41.802410 16913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:15:41.803932 16913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:15:41.804031 16913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:41.804133 16913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:15:41.804201 16913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:41.836030 16913 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:36949
I20260812 01:15:41.836107 17134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:36949 every 8 connection(s)
I20260812 01:15:41.837479 16913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:15:41.844884 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 16913
I20260812 01:15:41.845701 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:35213
--local_ip_for_outbound_sockets=127.20.151.3
--tserver_master_addrs=127.20.151.62:36143
--webserver_port=37107
--webserver_interface=127.20.151.3
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:15:41 dist-test-slave-21b9 krb5kdc[15666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497341, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:15:41.857577 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:41.840119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:58891 (local address 127.20.151.62:36143)
0812 01:15:41.840298 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:41.840303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:41.841442 (+ 1139us) server_negotiation.cc:407] Connection header received
0812 01:15:41.843299 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:41.843306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:41.843386 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:41.843473 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:41.845248 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:41.846166 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:41.847783 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:41.848269 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:41.848379 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:41.851942 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:41.851949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:41.851952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:41.851987 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:41.854521 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:41.855244 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:41.855248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:41.855250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:41.855399 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:41.855817 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:41.855822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:41.855826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:41.856123 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:41.856256 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:41.856951 (+ 695us) server_negotiation.cc:299] Negotiation successful
0812 01:15:41.857225 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 01:15:41.858628 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:41.840552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:58891)
0812 01:15:41.841264 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:41.841308 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:41.843024 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:41.843691 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:41.843699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:41.844239 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:15:41.845046 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:41.845074 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:41.846331 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:41.846335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:41.847633 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:41.847647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:41.848494 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:41.848502 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:41.848732 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:41.849406 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:15:41.849429 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:15:41.851699 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:15:41.854656 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:41.854665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:41.854683 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:41.855127 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:41.855506 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:41.855511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:41.855515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:41.855711 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:41.856257 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:15:41.856265 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:15:41.856603 (+ 338us) client_negotiation.cc:769] Sending connection context
0812 01:15:41.856852 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 01:15:41.857419 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":269,"threads_started":1}
I20260812 01:15:41.860211 17138 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36143
I20260812 01:15:41.860556 17138 heartbeater.cc:464] Registering TS with master...
I20260812 01:15:41.861423 17138 heartbeater.cc:511] Master 127.20.151.62:36143 requested a full tablet report, sending...
I20260812 01:15:41.863188 16597 ts_manager.cc:194] Registered new tserver with Master: eea84dad4cde44079f02ff13631874ff (127.20.151.2:36949)
I20260812 01:15:41.864317 16597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:58891
W20260812 01:15:42.200268 17145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:15:42.200596 17145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:15:42.200745 17145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:15:42.208563 17145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:15:42.208706 17145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:15:42.208858 17145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:15:42.219127 17145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:35213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=37107
--tserver_master_addrs=127.20.151.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:15:42.220569 17145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:15:42.221982 17145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:15:42.235556 17154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:42.235591 17153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:15:42.235728 17161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:15:42.236183 17145 server_base.cc:1034] running on GCE node
I20260812 01:15:42.236837 17145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:15:42.237900 17145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:15:42.239154 17145 hybrid_clock.cc:648] HybridClock initialized: now 1786497342239090 us; error 41 us; skew 500 ppm
I20260812 01:15:42.239578 17145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:15:42 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497342, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:15:42.242990 17145 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:15:42.244534 17145 webserver.cc:460] Webserver started at http://127.20.151.3:37107/ using document root <none> and password file <none>
I20260812 01:15:42.245287 17145 fs_manager.cc:359] Metadata directory not provided
I20260812 01:15:42.245431 17145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:15:42.251077 17145 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 01:15:42.253963 17173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:42.255398 17145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:15:42.255569 17145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "6c48c0cf90d0486aba167ad4b036c84e"
format_stamp: "Formatted at 2026-08-12 01:15:19 on dist-test-slave-21b9"
server_key: "77fa057286db48314f5ec2c63e9795e7"
server_key_iv: "c1d6a54ad6fbefd6c67b6dae53650855"
server_key_version: "encryptionkey@0"
I20260812 01:15:42.256642 17145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:15:42.277335 17145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:15:42.281505 17145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:15:42.281760 17145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:15:42.282815 17145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:15:42.284302 17145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:15:42.284408 17145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:42.284523 17145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:15:42.284593 17145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:15:42.316900 17145 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:35213
I20260812 01:15:42.316962 17346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:35213 every 8 connection(s)
I20260812 01:15:42.318358 17145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:15:42.322443 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 17145
Aug 12 01:15:42 dist-test-slave-21b9 krb5kdc[15666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497342, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:15:42.340408 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.320963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:36931 (local address 127.20.151.62:36143)
0812 01:15:42.321165 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.321171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.322435 (+ 1264us) server_negotiation.cc:407] Connection header received
0812 01:15:42.324817 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.324823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.324895 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.324983 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:42.326508 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.327236 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.329655 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.330099 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.330256 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.334445 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:42.334450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:42.334453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:42.334482 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:42.337322 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:42.338129 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:42.338134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:42.338138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:42.338290 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:42.338736 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:42.338742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:42.338746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:42.339128 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:42.339301 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:42.339983 (+ 682us) server_negotiation.cc:299] Negotiation successful
0812 01:15:42.340152 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 01:15:42.341810 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.321434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:36931)
0812 01:15:42.322233 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:42.322297 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:42.324492 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:42.325258 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:42.325266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:42.325788 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:15:42.326341 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.326358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.327486 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.327497 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.329508 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.329524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.330371 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.330376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.330584 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.331255 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:15:42.331313 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:15:42.334243 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:15:42.337483 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:42.337520 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:42.337538 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:42.338001 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:42.338396 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:15:42.338401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:15:42.338404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:15:42.338623 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:15:42.339248 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:15:42.339254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:15:42.339628 (+ 374us) client_negotiation.cc:769] Sending connection context
0812 01:15:42.339923 (+ 295us) client_negotiation.cc:240] Negotiation successful
0812 01:15:42.340458 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":279,"threads_started":1}
I20260812 01:15:42.342664 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.323426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.20.151.62:36143)
0812 01:15:42.323891 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.323898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.327918 (+ 4020us) server_negotiation.cc:407] Connection header received
0812 01:15:42.328093 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.328097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.328154 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.328231 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:42.330160 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.331103 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.332988 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.333607 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.333711 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.336716 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:42.336723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:42.336728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:42.336783 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:42.339700 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:42.340575 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:42.340580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:42.340583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:42.340724 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:42.341209 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:42.341213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:42.341215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:42.341497 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:42.341731 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:42.342107 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 01:15:42.342285 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":275,"threads_started":1}
I20260812 01:15:42.343410 17348 heartbeater.cc:347] Connected to a master server at 127.20.151.62:36143
I20260812 01:15:42.343830 17348 heartbeater.cc:464] Registering TS with master...
I20260812 01:15:42.344676 17348 heartbeater.cc:511] Master 127.20.151.62:36143 requested a full tablet report, sending...
I20260812 01:15:42.346221 16597 ts_manager.cc:194] Registered new tserver with Master: 6c48c0cf90d0486aba167ad4b036c84e (127.20.151.3:35213)
I20260812 01:15:42.347043 16597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:36931
I20260812 01:15:42.357491 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:15:42.364190 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.358452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.20.151.62:36143)
0812 01:15:42.358647 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.358654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.359101 (+ 447us) server_negotiation.cc:407] Connection header received
0812 01:15:42.359351 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.359355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.359499 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.359635 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:15:42.360096 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.360817 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.362263 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.362673 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.362785 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.363327 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:15:42.363552 (+ 225us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 01:15:42.363909 (+ 357us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60560: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 01:15:42.364297 17352 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60560: token expired
W20260812 01:15:42.364774 16483 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.151.62:36143)
I20260812 01:15:42.378780 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.365285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.20.151.62:36143)
0812 01:15:42.365437 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.365441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.365457 (+ 16us) server_negotiation.cc:407] Connection header received
0812 01:15:42.365495 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.365498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.365586 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.365659 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:42.367644 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.368350 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.370231 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.370668 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.370759 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.373407 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:42.373412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:42.373415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:42.373447 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:42.376128 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:42.376809 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:42.376814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:42.376816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:42.376948 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:42.377314 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:42.377317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:42.377319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:42.377675 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:42.377906 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:42.378235 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 01:15:42.378460 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 01:15:42.382303 16597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:15:42.389729 16597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:15:42.407665 16908 heartbeater.cc:503] Master 127.20.151.62:36143 was elected leader, sending a full tablet report...
I20260812 01:15:42.414430 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.402171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.20.151.2:36949)
0812 01:15:42.403752 (+ 1581us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.403758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.404395 (+ 637us) server_negotiation.cc:407] Connection header received
0812 01:15:42.404710 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.404718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.404975 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.405161 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:15:42.406284 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.407954 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.410862 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.412501 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.412660 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.412883 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:42.413890 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0812 01:15:42.414081 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1453,"thread_start_us":1316,"threads_started":1}
I20260812 01:15:42.414564 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.401635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:36949 (local address 127.0.0.1:35042)
0812 01:15:42.404227 (+ 2592us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:42.404286 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:42.404470 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:42.405495 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:42.405500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:42.405534 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:15:42.406044 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.406057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.408138 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.408142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.410658 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.410676 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.413087 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.413096 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.413722 (+ 626us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.413754 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:15:42.413834 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 01:15:42.414058 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2440,"mutex_wait_us":352,"spinlock_wait_cycles":1280,"thread_start_us":3230,"threads_started":1}
I20260812 01:15:42.415310 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.401607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41059 (local address 127.0.0.1:39080)
0812 01:15:42.404230 (+ 2623us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:42.404287 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:42.404470 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:42.405477 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:42.405486 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:42.405655 (+ 169us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:15:42.406141 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.406156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.408339 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.408346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.410774 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.410792 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.413134 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.413142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.413396 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.413462 (+ 66us) client_negotiation.cc:769] Sending connection context
0812 01:15:42.414724 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0812 01:15:42.414855 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2396,"mutex_wait_us":2,"spinlock_wait_cycles":14848,"thread_start_us":2220,"threads_started":1}
I20260812 01:15:42.415970 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.401795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.20.151.1:41059)
0812 01:15:42.403452 (+ 1657us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.403462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.404667 (+ 1205us) server_negotiation.cc:407] Connection header received
0812 01:15:42.404775 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.404783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.405011 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.405202 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:15:42.406357 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.408150 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.410976 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.412563 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.412716 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.412954 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:42.415238 (+ 2284us) server_negotiation.cc:299] Negotiation successful
0812 01:15:42.415621 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1482,"thread_start_us":1342,"threads_started":1}
I20260812 01:15:42.418996 17027 tablet_service.cc:1467] Processing CreateTablet for tablet ee0395c113c74fc8a8deb18775891915 (DEFAULT_TABLE table=test-table [id=2f0a0d2dd4fd4580972a4cba180b3244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:15:42.419179 16799 tablet_service.cc:1467] Processing CreateTablet for tablet ee0395c113c74fc8a8deb18775891915 (DEFAULT_TABLE table=test-table [id=2f0a0d2dd4fd4580972a4cba180b3244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:15:42.420567 16799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0395c113c74fc8a8deb18775891915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:15:42.420594 17027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0395c113c74fc8a8deb18775891915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:15:42.423409 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.404748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:35213 (local address 127.0.0.1:55288)
0812 01:15:42.415537 (+ 10789us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:42.415559 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:42.415694 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:42.416734 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:42.416738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:42.416765 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:15:42.416934 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.416945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.418984 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.418990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.420860 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.420878 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.422610 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.422617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.422810 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.422847 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 01:15:42.422944 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 01:15:42.423079 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10685,"thread_start_us":11355,"threads_started":1}
I20260812 01:15:42.423555 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.405115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55288 (local address 127.20.151.3:35213)
0812 01:15:42.415099 (+ 9984us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.415108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.416170 (+ 1062us) server_negotiation.cc:407] Connection header received
0812 01:15:42.416271 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.416277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.416525 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.416726 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:15:42.417153 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.418803 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.421035 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.422425 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.422571 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.422802 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:42.422999 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 01:15:42.423160 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8550,"thread_start_us":8412,"threads_started":1}
I20260812 01:15:42.427594 17244 tablet_service.cc:1467] Processing CreateTablet for tablet ee0395c113c74fc8a8deb18775891915 (DEFAULT_TABLE table=test-table [id=2f0a0d2dd4fd4580972a4cba180b3244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:15:42.429198 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0395c113c74fc8a8deb18775891915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:15:42.433892 17368 tablet_bootstrap.cc:492] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: Bootstrap starting.
I20260812 01:15:42.435907 17370 tablet_bootstrap.cc:492] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: Bootstrap starting.
I20260812 01:15:42.438351 17368 tablet_bootstrap.cc:654] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: Neither blocks nor log segments found. Creating new log.
I20260812 01:15:42.439487 17370 tablet_bootstrap.cc:654] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: Neither blocks nor log segments found. Creating new log.
I20260812 01:15:42.439734 17368 log.cc:824] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: Log is configured to *not* fsync() on all Append() calls
I20260812 01:15:42.440742 17370 log.cc:824] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: Log is configured to *not* fsync() on all Append() calls
I20260812 01:15:42.443470 17368 tablet_bootstrap.cc:492] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: No bootstrap required, opened a new log
I20260812 01:15:42.443820 17370 tablet_bootstrap.cc:492] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: No bootstrap required, opened a new log
I20260812 01:15:42.443833 17368 ts_tablet_manager.cc:1397] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260812 01:15:42.443948 17374 tablet_bootstrap.cc:492] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: Bootstrap starting.
I20260812 01:15:42.444156 17370 ts_tablet_manager.cc:1397] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260812 01:15:42.446866 17374 tablet_bootstrap.cc:654] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: Neither blocks nor log segments found. Creating new log.
I20260812 01:15:42.447912 17374 log.cc:824] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: Log is configured to *not* fsync() on all Append() calls
I20260812 01:15:42.450455 17374 tablet_bootstrap.cc:492] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: No bootstrap required, opened a new log
I20260812 01:15:42.450718 17374 ts_tablet_manager.cc:1397] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260812 01:15:42.454401 17368 raft_consensus.cc:348] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:42.454401 17370 raft_consensus.cc:348] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:42.454866 17368 raft_consensus.cc:374] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:15:42.454865 17370 raft_consensus.cc:374] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:15:42.454968 17370 raft_consensus.cc:729] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea84dad4cde44079f02ff13631874ff, State: Initialized, Role: FOLLOWER
I20260812 01:15:42.455019 17368 raft_consensus.cc:729] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758d529f16b8441f9d862de9f7138ca0, State: Initialized, Role: FOLLOWER
I20260812 01:15:42.455667 17368 consensus_queue.cc:260] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:42.455735 17370 consensus_queue.cc:260] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:42.459827 17374 raft_consensus.cc:348] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:42.460345 17374 raft_consensus.cc:374] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:15:42.460500 17374 raft_consensus.cc:729] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c48c0cf90d0486aba167ad4b036c84e, State: Initialized, Role: FOLLOWER
I20260812 01:15:42.461146 17374 consensus_queue.cc:260] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:42.461848 17368 ts_tablet_manager.cc:1428] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: Time spent starting tablet: real 0.018s user 0.008s sys 0.009s
I20260812 01:15:42.462929 17138 heartbeater.cc:503] Master 127.20.151.62:36143 was elected leader, sending a full tablet report...
I20260812 01:15:42.463644 17370 ts_tablet_manager.cc:1428] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: Time spent starting tablet: real 0.019s user 0.010s sys 0.009s
I20260812 01:15:42.464798 17348 heartbeater.cc:503] Master 127.20.151.62:36143 was elected leader, sending a full tablet report...
I20260812 01:15:42.465468 17374 ts_tablet_manager.cc:1428] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
W20260812 01:15:42.570063 17349 tablet.cc:2369] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:15:42.589378 17140 tablet.cc:2369] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:15:42.620421 16909 tablet.cc:2369] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:15:42.642659 17379 raft_consensus.cc:482] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:15:42.642953 17379 raft_consensus.cc:504] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:42.644793 17379 leader_election.cc:288] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eea84dad4cde44079f02ff13631874ff (127.20.151.2:36949), 6c48c0cf90d0486aba167ad4b036c84e (127.20.151.3:35213)
I20260812 01:15:42.651075 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.645267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:51305 (local address 127.20.151.2:36949)
0812 01:15:42.645746 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.645751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.645957 (+ 206us) server_negotiation.cc:407] Connection header received
0812 01:15:42.646170 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.646174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.646235 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.646310 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:15:42.646865 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.647635 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.649419 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.650197 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.650347 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.650421 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:42.650703 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 01:15:42.650827 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
I20260812 01:15:42.651201 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.645012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:36949 (local address 127.20.151.1:51305)
0812 01:15:42.645843 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:42.645859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:42.646016 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:42.646340 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:42.646344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:42.646371 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:15:42.646722 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.646731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.647768 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.647771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.649269 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.649282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.650401 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.650407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.650553 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.650581 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 01:15:42.650662 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 01:15:42.650787 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":536,"threads_started":1}
I20260812 01:15:42.651255 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.645381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:44547 (local address 127.20.151.3:35213)
0812 01:15:42.645596 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.645602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.646188 (+ 586us) server_negotiation.cc:407] Connection header received
0812 01:15:42.646350 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.646354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.646417 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.646490 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:15:42.647086 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.647834 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.649589 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.650376 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.650521 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.650594 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:42.650880 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 01:15:42.651012 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 01:15:42.651321 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.645278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:35213 (local address 127.20.151.1:44547)
0812 01:15:42.646070 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:42.646089 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:15:42.646215 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:15:42.646536 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:15:42.646540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:15:42.646565 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:15:42.646926 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.646936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.647968 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.647973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.649433 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:15:42.649446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.650595 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.650601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:15:42.650727 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.650751 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 01:15:42.650830 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 01:15:42.650954 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":486,"threads_started":1}
I20260812 01:15:42.652349 17061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0395c113c74fc8a8deb18775891915" candidate_uuid: "758d529f16b8441f9d862de9f7138ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea84dad4cde44079f02ff13631874ff" is_pre_election: true
I20260812 01:15:42.652346 17272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0395c113c74fc8a8deb18775891915" candidate_uuid: "758d529f16b8441f9d862de9f7138ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c48c0cf90d0486aba167ad4b036c84e" is_pre_election: true
I20260812 01:15:42.652841 17272 raft_consensus.cc:2454] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758d529f16b8441f9d862de9f7138ca0 in term 0.
I20260812 01:15:42.652837 17061 raft_consensus.cc:2454] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758d529f16b8441f9d862de9f7138ca0 in term 0.
I20260812 01:15:42.653476 16734 leader_election.cc:302] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6c48c0cf90d0486aba167ad4b036c84e, 758d529f16b8441f9d862de9f7138ca0, eea84dad4cde44079f02ff13631874ff; no voters:
I20260812 01:15:42.653846 17379 raft_consensus.cc:2781] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:15:42.653980 17379 raft_consensus.cc:482] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:15:42.654080 17379 raft_consensus.cc:3037] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:15:42.656389 17379 raft_consensus.cc:504] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:42.656960 17379 leader_election.cc:288] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [CANDIDATE]: Term 1 election: Requested vote from peers eea84dad4cde44079f02ff13631874ff (127.20.151.2:36949), 6c48c0cf90d0486aba167ad4b036c84e (127.20.151.3:35213)
I20260812 01:15:42.657408 17061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0395c113c74fc8a8deb18775891915" candidate_uuid: "758d529f16b8441f9d862de9f7138ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea84dad4cde44079f02ff13631874ff"
I20260812 01:15:42.657413 17272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0395c113c74fc8a8deb18775891915" candidate_uuid: "758d529f16b8441f9d862de9f7138ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c48c0cf90d0486aba167ad4b036c84e"
I20260812 01:15:42.657578 17272 raft_consensus.cc:3037] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:15:42.657588 17061 raft_consensus.cc:3037] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:15:42.659821 17061 raft_consensus.cc:2454] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758d529f16b8441f9d862de9f7138ca0 in term 1.
I20260812 01:15:42.659830 17272 raft_consensus.cc:2454] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758d529f16b8441f9d862de9f7138ca0 in term 1.
I20260812 01:15:42.660244 16733 leader_election.cc:302] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c48c0cf90d0486aba167ad4b036c84e, 758d529f16b8441f9d862de9f7138ca0; no voters:
I20260812 01:15:42.660574 17379 raft_consensus.cc:2781] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:15:42.661130 17379 raft_consensus.cc:686] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 1 LEADER]: Becoming Leader. State: Replica: 758d529f16b8441f9d862de9f7138ca0, State: Running, Role: LEADER
I20260812 01:15:42.661761 17379 consensus_queue.cc:237] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } } peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } }
I20260812 01:15:42.666252 16593 catalog_manager.cc:5495] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 758d529f16b8441f9d862de9f7138ca0 (127.20.151.1). New cstate: current_term: 1 leader_uuid: "758d529f16b8441f9d862de9f7138ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758d529f16b8441f9d862de9f7138ca0" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 } health_report { overall_health: UNKNOWN } } }
I20260812 01:15:42.709494 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.704004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.20.151.1:41059)
0812 01:15:42.704216 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.704222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.704240 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:15:42.704433 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.704438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.704545 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.704676 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:15:42.705062 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.705795 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.707247 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.707713 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.707825 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.708371 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:15:42.708881 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:15:42.709020 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:42.709128 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 01:15:42.709227 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 01:15:42.718058 17061 raft_consensus.cc:1261] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff [term 1 FOLLOWER]: Refusing update from remote peer 758d529f16b8441f9d862de9f7138ca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:15:42.718056 17272 raft_consensus.cc:1261] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e [term 1 FOLLOWER]: Refusing update from remote peer 758d529f16b8441f9d862de9f7138ca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:15:42.719218 17387 consensus_queue.cc:1035] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c48c0cf90d0486aba167ad4b036c84e" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 35213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:15:42.719624 17379 consensus_queue.cc:1035] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea84dad4cde44079f02ff13631874ff" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:15:42.739428 17391 mvcc.cc:205] Tried to move back new op lower bound from 7317493115762470912 to 7317493115546292224. Current Snapshot: MvccSnapshot[applied={T|T < 7317493115762470912}]
I20260812 01:15:42.753932 16595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:15:42.754498 16595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:15:42.759151 16595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Sending DeleteTablet for 3 replicas of tablet ee0395c113c74fc8a8deb18775891915
I20260812 01:15:42.760490 17027 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0395c113c74fc8a8deb18775891915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:15:42 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:35042
I20260812 01:15:42.760538 17023 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0395c113c74fc8a8deb18775891915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:15:42 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:35042
I20260812 01:15:42.760721 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 16664
I20260812 01:15:42.760977 16799 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0395c113c74fc8a8deb18775891915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:15:42 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:39080
I20260812 01:15:42.761723 17244 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0395c113c74fc8a8deb18775891915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:15:42 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:55288
I20260812 01:15:42.762575 17415 tablet_replica.cc:330] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: stopping tablet replica
I20260812 01:15:42.763089 17415 raft_consensus.cc:2229] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:15:42.763729 17415 raft_consensus.cc:2258] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:15:42.766067 17415 ts_tablet_manager.cc:1905] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:15:42.771426 17414 tablet_replica.cc:330] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: stopping tablet replica
I20260812 01:15:42.772110 17414 raft_consensus.cc:2229] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:15:42.772944 17414 raft_consensus.cc:2258] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:15:42.773348 17415 ts_tablet_manager.cc:1918] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:15:42.773563 17415 log.cc:1196] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/ee0395c113c74fc8a8deb18775891915
I20260812 01:15:42.774199 17415 ts_tablet_manager.cc:1939] T ee0395c113c74fc8a8deb18775891915 P 6c48c0cf90d0486aba167ad4b036c84e: Deleting consensus metadata
I20260812 01:15:42.775743 17414 ts_tablet_manager.cc:1905] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:15:42.776365 16574 catalog_manager.cc:4841] TS 6c48c0cf90d0486aba167ad4b036c84e (127.20.151.3:35213): tablet ee0395c113c74fc8a8deb18775891915 (table test-table [id=2f0a0d2dd4fd4580972a4cba180b3244]) successfully deleted
I20260812 01:15:42.781276 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.762365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.20.151.1:41059)
0812 01:15:42.762545 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.762552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.762572 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:15:42.762628 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.762634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.762709 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.762804 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:42.764883 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.766053 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.767984 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.768689 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.768811 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.772746 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:42.772764 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:42.772788 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:42.772842 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:42.776726 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:42.777623 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:42.777632 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:42.777636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:42.777926 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:42.778444 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:42.778449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:42.778453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:42.779029 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:42.779405 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:42.780629 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0812 01:15:42.780888 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 01:15:42.782393 16899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:15:42.782667 17409 tablet_replica.cc:330] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: stopping tablet replica
I20260812 01:15:42.783226 17409 raft_consensus.cc:2229] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:15:42.784138 17409 raft_consensus.cc:2258] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:15:42.787034 17409 ts_tablet_manager.cc:1905] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:15:42.794492 17409 ts_tablet_manager.cc:1918] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:15:42.794697 17409 log.cc:1196] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/ee0395c113c74fc8a8deb18775891915
I20260812 01:15:42.795261 17409 ts_tablet_manager.cc:1939] T ee0395c113c74fc8a8deb18775891915 P eea84dad4cde44079f02ff13631874ff: Deleting consensus metadata
I20260812 01:15:42.797385 16575 catalog_manager.cc:4841] TS eea84dad4cde44079f02ff13631874ff (127.20.151.2:36949): tablet ee0395c113c74fc8a8deb18775891915 (table test-table [id=2f0a0d2dd4fd4580972a4cba180b3244]) successfully deleted
W20260812 01:15:42.797928 16575 catalog_manager.cc:4808] TS eea84dad4cde44079f02ff13631874ff (127.20.151.2:36949): delete failed for tablet ee0395c113c74fc8a8deb18775891915 because the tablet was not found. No further retry: Not found: Tablet not found: ee0395c113c74fc8a8deb18775891915
I20260812 01:15:42.883003 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 16664
I20260812 01:15:42.908375 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 16913
W20260812 01:15:42.908299 16574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.151.1:41059 (error 108)
I20260812 01:15:42.910100 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.909610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41059 (local address 127.0.0.1:39096)
0812 01:15:42.909788 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:42.909919 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 01:15:42.910370 16574 catalog_manager.cc:4584] TS 758d529f16b8441f9d862de9f7138ca0 (127.20.151.1:41059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee0395c113c74fc8a8deb18775891915: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41059: connect: Connection refused (error 111)
I20260812 01:15:42.923019 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.909134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.20.151.2:36949)
0812 01:15:42.909325 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:42.909330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:42.909429 (+ 99us) server_negotiation.cc:407] Connection header received
0812 01:15:42.909614 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:42.909620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:42.909714 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:42.909816 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:42.911473 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.912248 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.913478 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:42.913889 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:42.913982 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:42.916925 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:42.916942 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:42.916959 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:42.916999 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:42.920026 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:42.920889 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:42.920899 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:42.920901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:42.921056 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:42.921409 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:42.921415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:42.921418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:42.921884 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:42.922129 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:42.922508 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 01:15:42.922689 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:15:42.923745 17130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:15:42.963999 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.963514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41059 (local address 127.0.0.1:39112)
0812 01:15:42.963739 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:42.963854 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 01:15:42.965040 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:42.964653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41059 (local address 127.0.0.1:39124)
0812 01:15:42.964804 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:42.964895 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 01:15:43.015702 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 16913
I20260812 01:15:43.026471 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:43.025944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41059 (local address 127.0.0.1:39132)
0812 01:15:43.026174 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:43.026295 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 01:15:43.027650 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:43.027181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41059 (local address 127.0.0.1:39148)
0812 01:15:43.027380 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:43.027467 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 01:15:43.044075 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 17145
I20260812 01:15:43.058236 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:43.044824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.20.151.3:35213)
0812 01:15:43.044985 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 01:15:43.044990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:15:43.045009 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:15:43.045156 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:15:43.045160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:15:43.045231 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:15:43.045321 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:15:43.046623 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:43.047400 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:43.048814 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:15:43.049245 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:15:43.049346 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:15:43.052525 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:15:43.052543 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:15:43.052562 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:15:43.052606 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:15:43.055608 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:43.056303 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:43.056311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:43.056313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:43.056474 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:15:43.056865 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:15:43.056869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:15:43.056871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:15:43.057258 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:15:43.057487 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 01:15:43.057796 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 01:15:43.057966 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 01:15:43.058923 17339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:15:43.108333 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:43.107847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41059 (local address 127.0.0.1:39152)
0812 01:15:43.108068 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:43.108172 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 01:15:43.109531 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:43.109023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41059 (local address 127.0.0.1:39154)
0812 01:15:43.109293 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:43.109370 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260812 01:15:43.148511 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 17145
I20260812 01:15:43.170092 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 16552
I20260812 01:15:43.171351 16652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:15:43.287758 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 16552
I20260812 01:15:43.287837 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:15:43.287142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41059 (local address 127.0.0.1:39156)
0812 01:15:43.287580 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0812 01:15:43.287685 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269}
I20260812 01:16:03.323772 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:36143
--webserver_interface=127.20.151.62
--webserver_port=35579
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:36143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:03.677326 17423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:03.677662 17423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:03.677796 17423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:03.685302 17423 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 01:16:03.685426 17423 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 01:16:03.685503 17423 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 01:16:03.685573 17423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:03.685648 17423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:03.685745 17423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:03.685809 17423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:03.685886 17423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:03.696192 17423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:36143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:36143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=35579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:03.697577 17423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:03.698935 17423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:03.710886 17432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:03.711009 17440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:03.710916 17433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:03.711843 17423 server_base.cc:1034] running on GCE node
I20260812 01:16:03.712677 17423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:03.714140 17423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:03.715406 17423 hybrid_clock.cc:648] HybridClock initialized: now 1786497363715331 us; error 38 us; skew 500 ppm
I20260812 01:16:03.715771 17423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:03 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497363, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:03.722427 17423 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:16:03.724048 17423 webserver.cc:460] Webserver started at http://127.20.151.62:35579/ using document root <none> and password file <none>
I20260812 01:16:03.724797 17423 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:03.724910 17423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:03.730716 17423 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 01:16:03.733880 17451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:03.735378 17423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 01:16:03.735563 17423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "ffe0e4afbfc54ebcb69b902b82d637ca"
format_stamp: "Formatted at 2026-08-12 01:15:17 on dist-test-slave-21b9"
server_key: "f9a7ae0c2ce22fdaceeaed83ea66e8c0"
server_key_iv: "8468ba1a4198d9ae383942ca1a0b1e8f"
server_key_version: "encryptionkey@0"
I20260812 01:16:03.736548 17423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:03.762362 17423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:03.766803 17423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:03.767092 17423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:03.786262 17423 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:36143
I20260812 01:16:03.786324 17537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:36143 every 8 connection(s)
I20260812 01:16:03.787901 17423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:03.789368 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 17423
I20260812 01:16:03.790539 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:41059
--local_ip_for_outbound_sockets=127.20.151.1
--tserver_master_addrs=127.20.151.62:36143
--webserver_port=41557
--webserver_interface=127.20.151.1
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:16:03.797290 17539 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:16:03.802081 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Bootstrap starting.
I20260812 01:16:03.825137 17539 log.cc:824] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:03.843554 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:16:03.844291 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Bootstrap complete.
I20260812 01:16:03.855952 17539 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:16:03.856938 17539 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe0e4afbfc54ebcb69b902b82d637ca, State: Initialized, Role: FOLLOWER
I20260812 01:16:03.857596 17539 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:16:03.857832 17539 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:03.857951 17539 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:03.858121 17539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 2 FOLLOWER]: Advancing to term 3
I20260812 01:16:03.862093 17539 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:16:03.862577 17539 leader_election.cc:302] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffe0e4afbfc54ebcb69b902b82d637ca; no voters:
I20260812 01:16:03.863248 17539 leader_election.cc:288] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 01:16:03.863411 17545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 3 FOLLOWER]: Leader election won for term 3
I20260812 01:16:03.864564 17545 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [term 3 LEADER]: Becoming Leader. State: Replica: ffe0e4afbfc54ebcb69b902b82d637ca, State: Running, Role: LEADER
I20260812 01:16:03.865190 17545 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } }
I20260812 01:16:03.865507 17539 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:03.871371 17547 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } } }
I20260812 01:16:03.871982 17547 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:03.872608 17548 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffe0e4afbfc54ebcb69b902b82d637ca. Latest consensus state: current_term: 3 leader_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffe0e4afbfc54ebcb69b902b82d637ca" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 36143 } } }
I20260812 01:16:03.872954 17548 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:03.881934 17561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:03.895236 17561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:03.896987 17561 catalog_manager.cc:1259] Loaded cluster ID: 4080295ec7704313968f23d26eb83683
I20260812 01:16:03.897541 17561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:03.901785 17561 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:03.903827 17561 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Loaded TSK: 0
I20260812 01:16:03.904089 17561 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffe0e4afbfc54ebcb69b902b82d637ca: Loaded TSK: 1
F20260812 01:16:03.905349 17561 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0e2ea689ad google::LogMessage::Fail() at ??:0
@ 0x7f0e2ea6b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f0e2ea684de google::LogMessage::Flush() at ??:0
@ 0x7f0e2ea69a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0e327cc7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0e327bb73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f0e444971c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0e44496783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0e445222a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f0e445220c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f0e4456c5e9 std::function<>::operator()() at ??:0
@ 0x7f0e4449acdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0e44498c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0e44520610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0e44520439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0e316b4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0e316b791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0e316b7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0e31692eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0e3496c6db start_thread at ??:0
@ 0x7f0e2d90261f clone at ??:0
W20260812 01:16:04.216161 17543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:04.216470 17543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:04.216594 17543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:04.224308 17543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:04.224434 17543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:04.224591 17543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:16:04.234795 17543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:41059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=41557
--tserver_master_addrs=127.20.151.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:04.236194 17543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:04.237562 17543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:04.250587 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:04.250633 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:04.250681 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:04.251959 17543 server_base.cc:1034] running on GCE node
I20260812 01:16:04.252605 17543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:04.253618 17543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:04.254828 17543 hybrid_clock.cc:648] HybridClock initialized: now 1786497364254766 us; error 57 us; skew 500 ppm
I20260812 01:16:04.255146 17543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:04 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497364, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:04.258296 17543 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:16:04.259845 17543 webserver.cc:460] Webserver started at http://127.20.151.1:41557/ using document root <none> and password file <none>
I20260812 01:16:04.260566 17543 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:04.260692 17543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:04.266211 17543 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 01:16:04.269387 17639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:04.270754 17543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 01:16:04.270906 17543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "758d529f16b8441f9d862de9f7138ca0"
format_stamp: "Formatted at 2026-08-12 01:15:18 on dist-test-slave-21b9"
server_key: "9575f37e102b2c18bd6aef86a7ef812b"
server_key_iv: "33aab32f960a8267f870d9f7f7811060"
server_key_version: "encryptionkey@0"
I20260812 01:16:04.271947 17543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:04.303610 17543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:04.307870 17543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:04.308141 17543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:04.309195 17543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:04.311339 17651 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 01:16:04.317154 17651 ts_tablet_manager.cc:1852] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260812 01:16:04.318161 17651 ts_tablet_manager.cc:1905] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:04.325742 17651 ts_tablet_manager.cc:1918] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:04.325935 17651 log.cc:1196] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/ee0395c113c74fc8a8deb18775891915
I20260812 01:16:04.326442 17651 ts_tablet_manager.cc:1939] T ee0395c113c74fc8a8deb18775891915 P 758d529f16b8441f9d862de9f7138ca0: Deleting consensus metadata
I20260812 01:16:04.326830 17543 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260812 01:16:04.326967 17543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260812 01:16:04.327091 17543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:04.327167 17543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:04.360361 17543 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:41059
I20260812 01:16:04.360467 17836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:41059 every 8 connection(s)
I20260812 01:16:04.361773 17543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:16:04.366743 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:04.364837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:49069)
0812 01:16:04.365615 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:04.365853 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":282,"threads_started":1}
W20260812 01:16:04.367401 17838 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:36143 (0 consecutive failures): Network error: Failed to ping master at 127.20.151.62:36143: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
I20260812 01:16:04.368209 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:04.367879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:52255)
0812 01:16:04.368030 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:04.368102 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 01:16:04.369131 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:04.368704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:38477)
0812 01:16:04.368867 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:04.368970 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 01:16:04.369410 17838 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 01:16:04.372288 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 17543
I20260812 01:16:04.372932 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:36949
--local_ip_for_outbound_sockets=127.20.151.2
--tserver_master_addrs=127.20.151.62:36143
--webserver_port=41589
--webserver_interface=127.20.151.2
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:04.728034 17842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:04.728384 17842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:04.728546 17842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:04.736035 17842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:04.736176 17842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:04.736338 17842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:16:04.746306 17842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:36949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=41589
--tserver_master_addrs=127.20.151.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:04.747714 17842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:04.749073 17842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:04.762631 17850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:04.762557 17851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:04.762611 17856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:04.763784 17842 server_base.cc:1034] running on GCE node
I20260812 01:16:04.764415 17842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:04.765425 17842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:04.766647 17842 hybrid_clock.cc:648] HybridClock initialized: now 1786497364766591 us; error 45 us; skew 500 ppm
I20260812 01:16:04.766973 17842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:04 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497364, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:04.770043 17842 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:16:04.771556 17842 webserver.cc:460] Webserver started at http://127.20.151.2:41589/ using document root <none> and password file <none>
I20260812 01:16:04.772244 17842 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:04.772370 17842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:04.778116 17842 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 01:16:04.780941 17864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:04.782290 17842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:16:04.782455 17842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "eea84dad4cde44079f02ff13631874ff"
format_stamp: "Formatted at 2026-08-12 01:15:18 on dist-test-slave-21b9"
server_key: "5c55df03227ba994b5c061f933caa1a0"
server_key_iv: "c5ab1135cd30d1a255e3413819e34b2c"
server_key_version: "encryptionkey@0"
I20260812 01:16:04.783542 17842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:04.796542 17842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:04.800828 17842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:04.801105 17842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:04.802121 17842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:04.803586 17842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:04.803699 17842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:04.803807 17842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:04.803882 17842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:04.837003 17842 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:36949
I20260812 01:16:04.837066 18050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:36949 every 8 connection(s)
I20260812 01:16:04.838493 17842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:16:04.843375 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:04.841478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:38235)
0812 01:16:04.842219 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:04.842454 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":286,"threads_started":1}
W20260812 01:16:04.844123 18053 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:36143 (0 consecutive failures): Network error: Failed to ping master at 127.20.151.62:36143: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
I20260812 01:16:04.844919 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:04.844612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:42731)
0812 01:16:04.844745 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:04.844814 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 01:16:04.845666 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:04.845376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:37485)
0812 01:16:04.845482 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:04.845552 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 01:16:04.845887 18053 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 01:16:04.848167 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 17842
I20260812 01:16:04.848753 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:35213
--local_ip_for_outbound_sockets=127.20.151.3
--tserver_master_addrs=127.20.151.62:36143
--webserver_port=37107
--webserver_interface=127.20.151.3
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:05.207208 18057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:05.207574 18057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:05.207729 18057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:05.215206 18057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:05.215385 18057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:05.215566 18057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:16:05.225858 18057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:35213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=37107
--tserver_master_addrs=127.20.151.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:05.227301 18057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:05.228734 18057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:05.242059 18062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:05.242158 18064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:05.242129 18067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:05.243052 18057 server_base.cc:1034] running on GCE node
I20260812 01:16:05.243755 18057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:05.244808 18057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:05.246029 18057 hybrid_clock.cc:648] HybridClock initialized: now 1786497365245973 us; error 46 us; skew 500 ppm
I20260812 01:16:05.246343 18057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:05 dist-test-slave-21b9 krb5kdc[15666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497365, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:05.249490 18057 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:16:05.251037 18057 webserver.cc:460] Webserver started at http://127.20.151.3:37107/ using document root <none> and password file <none>
I20260812 01:16:05.251798 18057 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:05.251932 18057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:05.257453 18057 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 01:16:05.260375 18079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:05.261740 18057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:16:05.261905 18057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "6c48c0cf90d0486aba167ad4b036c84e"
format_stamp: "Formatted at 2026-08-12 01:15:19 on dist-test-slave-21b9"
server_key: "77fa057286db48314f5ec2c63e9795e7"
server_key_iv: "c1d6a54ad6fbefd6c67b6dae53650855"
server_key_version: "encryptionkey@0"
I20260812 01:16:05.262928 18057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:05.281484 18057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:05.285859 18057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:05.286140 18057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:05.287322 18057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:05.288818 18057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:05.288930 18057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:05.289047 18057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:05.289119 18057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:05.322084 18057 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:35213
I20260812 01:16:05.322170 18267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:35213 every 8 connection(s)
I20260812 01:16:05.323580 18057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:16:05.324134 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 18057
I20260812 01:16:05.329595 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:05.327674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:42333)
0812 01:16:05.328421 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:05.328695 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":277,"threads_started":1}
W20260812 01:16:05.330351 18268 heartbeater.cc:644] Failed to heartbeat to 127.20.151.62:36143 (0 consecutive failures): Network error: Failed to ping master at 127.20.151.62:36143: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
I20260812 01:16:05.331207 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:05.330864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:49045)
0812 01:16:05.331020 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:05.331088 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 01:16:05.332042 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:05.331714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:48151)
0812 01:16:05.331858 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:05.331931 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 01:16:05.332271 18268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 01:16:05.371009 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:05.370006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:40443)
0812 01:16:05.370533 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:05.370691 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":261,"threads_started":1}
I20260812 01:16:05.847433 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:05.846551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:46421)
0812 01:16:05.847051 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:05.847140 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":242,"threads_started":1}
I20260812 01:16:06.333738 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:06.332883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:49165)
0812 01:16:06.333359 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:06.333472 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":228,"threads_started":1}
I20260812 01:16:06.372736 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:06.371980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:57299)
0812 01:16:06.372473 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:06.372567 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260812 01:16:06.849342 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:06.848497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:34435)
0812 01:16:06.849049 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:06.849161 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":288,"threads_started":1}
I20260812 01:16:07.335587 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:07.334735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:57375)
0812 01:16:07.335301 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:07.335392 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":303,"threads_started":1}
I20260812 01:16:07.374485 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:07.373681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:35053)
0812 01:16:07.374202 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:07.374297 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":268,"threads_started":1}
I20260812 01:16:07.851137 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:07.850354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:59039)
0812 01:16:07.850857 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:07.850955 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":274,"threads_started":1}
I20260812 01:16:08.337450 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:08.336612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:59847)
0812 01:16:08.337160 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:08.337255 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":278,"threads_started":1}
I20260812 01:16:08.376402 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:08.375551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:55401)
0812 01:16:08.376102 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:08.376199 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":281,"threads_started":1}
I20260812 01:16:08.852934 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:08.852136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:37473)
0812 01:16:08.852604 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:08.852705 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260812 01:16:09.339211 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:09.338425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:52073)
0812 01:16:09.338893 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:09.338993 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":233,"threads_started":1}
I20260812 01:16:09.378134 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:09.377383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:56807)
0812 01:16:09.377873 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:09.377959 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":261,"threads_started":1}
I20260812 01:16:09.854578 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:09.853815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:39075)
0812 01:16:09.854286 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:09.854372 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":238,"threads_started":1}
I20260812 01:16:10.341054 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:10.340221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:44519)
0812 01:16:10.340747 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:10.340853 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":262,"threads_started":1}
I20260812 01:16:10.379789 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:10.379024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:35421)
0812 01:16:10.379532 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:10.379619 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":277,"threads_started":1}
I20260812 01:16:10.856273 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:10.855502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:34807)
0812 01:16:10.855993 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:10.856093 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":231,"threads_started":1}
I20260812 01:16:11.343036 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:11.342151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:58013)
0812 01:16:11.342738 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:11.342847 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":289,"threads_started":1}
I20260812 01:16:11.381662 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:11.380739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:37605)
0812 01:16:11.381307 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:11.381404 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":254,"threads_started":1}
I20260812 01:16:11.858297 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:11.857324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:49533)
0812 01:16:11.857938 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:11.858044 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":257,"threads_started":1}
I20260812 01:16:12.345036 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:12.344154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:49979)
0812 01:16:12.344709 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:12.344807 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":261,"threads_started":1}
I20260812 01:16:12.383409 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:12.382582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:57381)
0812 01:16:12.383069 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:12.383162 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260812 01:16:12.860229 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:12.859359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:36057)
0812 01:16:12.859942 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:12.860036 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":339,"threads_started":1}
I20260812 01:16:13.346760 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:13.346007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:42397)
0812 01:16:13.346486 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:13.346577 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":234,"threads_started":1}
I20260812 01:16:13.387580 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:13.384272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:39491)
0812 01:16:13.387185 (+ 2913us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:13.387348 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2788,"thread_start_us":2662,"threads_started":1}
I20260812 01:16:13.861981 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:13.861177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:59143)
0812 01:16:13.861662 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:13.861764 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":232,"threads_started":1}
I20260812 01:16:14.348644 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:14.347814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:50591)
0812 01:16:14.348362 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:14.348451 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":265,"threads_started":1}
I20260812 01:16:14.389320 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:14.388566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:58375)
0812 01:16:14.389038 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:14.389130 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
I20260812 01:16:14.863790 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:14.863025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:49653)
0812 01:16:14.863527 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:14.863619 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":277,"threads_started":1}
I20260812 01:16:15.350463 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:15.349687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:44475)
0812 01:16:15.350194 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:15.350287 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":261,"threads_started":1}
I20260812 01:16:15.391175 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:15.390356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:52223)
0812 01:16:15.390867 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:15.390956 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":288,"threads_started":1}
I20260812 01:16:15.865379 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:15.864667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:39489)
0812 01:16:15.865100 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:15.865193 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":203,"threads_started":1}
I20260812 01:16:16.352274 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:16.351492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:42533)
0812 01:16:16.351965 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:16.352080 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":233,"threads_started":1}
I20260812 01:16:16.393146 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:16.392168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:44137)
0812 01:16:16.392879 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:16.392971 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":482,"threads_started":1}
I20260812 01:16:16.867162 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:16.866354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:58605)
0812 01:16:16.866893 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:16.866997 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":216,"threads_started":1}
I20260812 01:16:17.354106 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:17.353301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:52115)
0812 01:16:17.353848 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:17.353937 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":308,"threads_started":1}
I20260812 01:16:17.394951 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:17.394119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:34819)
0812 01:16:17.394639 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:17.394753 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":262,"threads_started":1}
I20260812 01:16:17.868922 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:17.868095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:44547)
0812 01:16:17.868598 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:17.868740 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":245,"threads_started":1}
I20260812 01:16:18.355949 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:18.355122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:32875)
0812 01:16:18.355645 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:18.355751 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":260,"threads_started":1}
I20260812 01:16:18.396831 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:18.395943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:53061)
0812 01:16:18.396497 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:18.396603 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":264,"threads_started":1}
I20260812 01:16:18.870733 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:18.869876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:42549)
0812 01:16:18.870410 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:18.870503 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":258,"threads_started":1}
I20260812 01:16:19.357908 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:19.356955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:35131)
0812 01:16:19.357555 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:19.357670 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":296,"threads_started":1}
I20260812 01:16:19.398932 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:19.398031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:51313)
0812 01:16:19.398609 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:19.398719 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":280,"threads_started":1}
I20260812 01:16:19.872736 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:19.871969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.2:40843)
0812 01:16:19.872415 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:19.872552 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":222,"threads_started":1}
I20260812 01:16:20.334064 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 17543
I20260812 01:16:20.352056 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:20.335176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.20.151.1:41059)
0812 01:16:20.335908 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:20.335930 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:20.335979 (+ 49us) server_negotiation.cc:407] Connection header received
0812 01:16:20.337902 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:20.337913 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:20.338347 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:20.338650 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:20.340167 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:20.341438 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:20.342661 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:20.343341 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:20.343470 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:20.346052 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:20.346060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:20.346076 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:20.346123 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:20.349023 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:20.349656 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:20.349663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:20.349669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:20.349834 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:20.350130 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:20.350132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:20.350134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:20.350681 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:20.350913 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:20.351323 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 01:16:20.351748 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":406,"threads_started":1}
I20260812 01:16:20.353423 17828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:20.359921 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:20.359160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.3:44565)
0812 01:16:20.359643 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:20.359736 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":245,"threads_started":1}
I20260812 01:16:20.441042 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:20.439934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:36143 (local address 127.20.151.1:34635)
0812 01:16:20.440508 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:20.440715 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.62:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":285,"threads_started":1}
I20260812 01:16:20.443434 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 17543
I20260812 01:16:20.476212 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 17842
I20260812 01:16:20.493973 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:20.477244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.20.151.2:36949)
0812 01:16:20.477752 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:20.477767 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:20.477804 (+ 37us) server_negotiation.cc:407] Connection header received
0812 01:16:20.479402 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:20.479413 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:20.479842 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:20.480172 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:20.481550 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:20.482834 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:20.484282 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:20.484964 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:20.485120 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:20.487903 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:20.487913 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:20.487929 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:20.488011 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:20.490909 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:20.491599 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:20.491606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:20.491613 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:20.491801 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:20.492130 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:20.492133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:20.492135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:20.492669 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:20.492921 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:20.493262 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 01:16:20.493679 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260812 01:16:20.495462 18044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:20.580852 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 17842
I20260812 01:16:20.609949 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 18057
I20260812 01:16:20.628685 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:20.611033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.20.151.3:35213)
0812 01:16:20.611682 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:20.611698 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:20.611746 (+ 48us) server_negotiation.cc:407] Connection header received
0812 01:16:20.613461 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:20.613471 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:20.613911 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:20.614259 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:20.615981 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:20.617235 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:20.618552 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:20.619250 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:20.619486 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:20.622137 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:20.622145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:20.622161 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:20.622210 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:20.625147 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:20.625872 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:20.625879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:20.625887 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:20.626070 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:20.626462 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:20.626466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:20.626469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:20.627023 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:20.627348 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:20.627925 (+ 577us) server_negotiation.cc:299] Negotiation successful
0812 01:16:20.628364 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":312,"threads_started":1}
I20260812 01:16:20.630131 18261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:20.716212 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 18057
2026-08-12T01:16:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65286 ms)
[----------] 30 tests from SecurityITest (314086 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:16:20 dist-test-slave-21b9 krb5kdc[18372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:16:20 dist-test-slave-21b9 krb5kdc[18372](info): set up 2 sockets
Aug 12 01:16:20 dist-test-slave-21b9 krb5kdc[18372](info): commencing operation
krb5kdc: starting...
W20260812 01:16:22.020188 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:16:22 dist-test-slave-21b9 krb5kdc[18372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:16:22.377228 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:34769
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:34769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:22.734120 18397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:22.734439 18397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:22.734571 18397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:22.742003 18397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:22.742151 18397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:22.742233 18397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:22.742311 18397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:22.742369 18397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:22.752503 18397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:34769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:34769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:22.753891 18397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:22.755218 18397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:22.767379 18404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:22.767345 18411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:22.767450 18403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:22.768038 18397 server_base.cc:1034] running on GCE node
I20260812 01:16:22.768851 18397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:22.770287 18397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:22.771500 18397 hybrid_clock.cc:648] HybridClock initialized: now 1786497382771443 us; error 34 us; skew 500 ppm
I20260812 01:16:22.771832 18397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:22 dist-test-slave-21b9 krb5kdc[18372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497382, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:22.779194 18397 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:16:22.780815 18397 webserver.cc:460] Webserver started at http://127.20.151.62:42031/ using document root <none> and password file <none>
I20260812 01:16:22.781554 18397 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:22.781682 18397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:22.781993 18397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:22.785101 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "663443db6f8c41da94af1e18599ace43"
format_stamp: "Formatted at 2026-08-12 01:16:22 on dist-test-slave-21b9"
server_key: "02f9313b90114b2b13f0ecf01667544c"
server_key_iv: "7a52ebf8d2cb95de41e411568d714960"
server_key_version: "encryptionkey@0"
I20260812 01:16:22.785849 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "663443db6f8c41da94af1e18599ace43"
format_stamp: "Formatted at 2026-08-12 01:16:22 on dist-test-slave-21b9"
server_key: "02f9313b90114b2b13f0ecf01667544c"
server_key_iv: "7a52ebf8d2cb95de41e411568d714960"
server_key_version: "encryptionkey@0"
I20260812 01:16:22.791220 18397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260812 01:16:22.794749 18421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:22.796139 18397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:16:22.796320 18397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "663443db6f8c41da94af1e18599ace43"
format_stamp: "Formatted at 2026-08-12 01:16:22 on dist-test-slave-21b9"
server_key: "02f9313b90114b2b13f0ecf01667544c"
server_key_iv: "7a52ebf8d2cb95de41e411568d714960"
server_key_version: "encryptionkey@0"
I20260812 01:16:22.796518 18397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:22.812784 18397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:22.817210 18397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:22.817483 18397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:22.836164 18397 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:34769
I20260812 01:16:22.836172 18512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:34769 every 8 connection(s)
I20260812 01:16:22.837639 18397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:22.841784 18514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:22.841831 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 18397
I20260812 01:16:22.842054 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:22.842916 21084 external_mini_cluster.cc:1428] Setting key 28d31b11ba3b610139dac6da3c4d7e66
I20260812 01:16:22.854876 18514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43: Bootstrap starting.
Aug 12 01:16:22 dist-test-slave-21b9 krb5kdc[18372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:22.858305 18514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:22.859462 18514 log.cc:824] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:22.862555 18514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43: No bootstrap required, opened a new log
I20260812 01:16:22.866096 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:22.845171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.20.151.62:34769)
0812 01:16:22.845988 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:22.846000 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:22.846044 (+ 44us) server_negotiation.cc:407] Connection header received
0812 01:16:22.848424 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:22.848450 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:22.848876 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:22.849274 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:22.850786 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:22.852190 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:22.853519 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:22.854212 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:22.854380 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:22.858756 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:22.858765 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:22.858781 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:22.858819 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:22.861971 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:22.862679 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:22.862687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:22.862697 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:22.862879 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:22.863313 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:22.863316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:22.863318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:22.863869 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:22.864169 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:22.864658 (+ 489us) server_negotiation.cc:299] Negotiation successful
0812 01:16:22.865080 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":356,"threads_started":1}
I20260812 01:16:22.870647 18514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663443db6f8c41da94af1e18599ace43" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34769 } }
I20260812 01:16:22.871011 18514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:22.871120 18514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663443db6f8c41da94af1e18599ace43, State: Initialized, Role: FOLLOWER
I20260812 01:16:22.871833 18514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663443db6f8c41da94af1e18599ace43" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34769 } }
I20260812 01:16:22.872114 18514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:22.872268 18514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:22.872445 18514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:22.874622 18514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663443db6f8c41da94af1e18599ace43" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34769 } }
I20260812 01:16:22.875188 18514 leader_election.cc:302] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 663443db6f8c41da94af1e18599ace43; no voters:
I20260812 01:16:22.875859 18514 leader_election.cc:288] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:22.875993 18523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:22.876698 18523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [term 1 LEADER]: Becoming Leader. State: Replica: 663443db6f8c41da94af1e18599ace43, State: Running, Role: LEADER
I20260812 01:16:22.877202 18523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663443db6f8c41da94af1e18599ace43" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34769 } }
I20260812 01:16:22.877959 18514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:22.880033 18528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 663443db6f8c41da94af1e18599ace43. Latest consensus state: current_term: 1 leader_uuid: "663443db6f8c41da94af1e18599ace43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663443db6f8c41da94af1e18599ace43" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34769 } } }
I20260812 01:16:22.880528 18528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:22.880110 18524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "663443db6f8c41da94af1e18599ace43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663443db6f8c41da94af1e18599ace43" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 34769 } } }
I20260812 01:16:22.880939 18524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:22.884503 18533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:22.890552 18533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:22.902530 18533 catalog_manager.cc:1347] Generated new cluster ID: 63274066e6f64b8496cd79e4124b30c7
I20260812 01:16:22.902693 18533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:22.920307 18533 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:22.922181 18533 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:22.931023 18533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43: Generated new TSK 0
I20260812 01:16:22.932128 18533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:16:23.011861 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34769
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:23.369800 18559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:23.370101 18559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:23.370241 18559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:23.377646 18559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:23.377764 18559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:23.377931 18559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:16:23.388145 18559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:23.389546 18559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:23.390942 18559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:23.404613 18564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:23.404656 18565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:23.404670 18567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:23.405462 18559 server_base.cc:1034] running on GCE node
I20260812 01:16:23.406121 18559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:23.407075 18559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:23.408283 18559 hybrid_clock.cc:648] HybridClock initialized: now 1786497383408224 us; error 53 us; skew 500 ppm
I20260812 01:16:23.408601 18559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:23 dist-test-slave-21b9 krb5kdc[18372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497383, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:23.411760 18559 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:16:23.413233 18559 webserver.cc:460] Webserver started at http://127.20.151.1:41397/ using document root <none> and password file <none>
I20260812 01:16:23.413928 18559 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:23.414049 18559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:23.414376 18559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:23.417409 18559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "7640324834eb427ea6fdd78ae6aa63b7"
format_stamp: "Formatted at 2026-08-12 01:16:23 on dist-test-slave-21b9"
server_key: "5f1803750bccf903eca29500f4943fe0"
server_key_iv: "2d9fa90324143e2f9681903385a82247"
server_key_version: "encryptionkey@0"
I20260812 01:16:23.418198 18559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "7640324834eb427ea6fdd78ae6aa63b7"
format_stamp: "Formatted at 2026-08-12 01:16:23 on dist-test-slave-21b9"
server_key: "5f1803750bccf903eca29500f4943fe0"
server_key_iv: "2d9fa90324143e2f9681903385a82247"
server_key_version: "encryptionkey@0"
I20260812 01:16:23.423497 18559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 01:16:23.427037 18580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:23.428467 18559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:16:23.428640 18559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "7640324834eb427ea6fdd78ae6aa63b7"
format_stamp: "Formatted at 2026-08-12 01:16:23 on dist-test-slave-21b9"
server_key: "5f1803750bccf903eca29500f4943fe0"
server_key_iv: "2d9fa90324143e2f9681903385a82247"
server_key_version: "encryptionkey@0"
I20260812 01:16:23.428838 18559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:23.453840 18559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:23.458384 18559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:23.458663 18559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:23.459676 18559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:23.461120 18559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:23.461196 18559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:23.461283 18559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:23.461328 18559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:23.492792 18559 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:34179
I20260812 01:16:23.492856 18780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:34179 every 8 connection(s)
I20260812 01:16:23.494087 18559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:16:23.498236 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 18559
I20260812 01:16:23.498473 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:16:23.499341 21084 external_mini_cluster.cc:1428] Setting key 7532295f21e6d329c688bf2adebe15ca
Aug 12 01:16:23 dist-test-slave-21b9 krb5kdc[18372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497383, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:23.516319 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:23.496680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:36729 (local address 127.20.151.62:34769)
0812 01:16:23.497135 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:23.497139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:23.498197 (+ 1058us) server_negotiation.cc:407] Connection header received
0812 01:16:23.500810 (+ 2613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:23.500816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:23.500898 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:23.501010 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:23.503177 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:23.503964 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:23.506031 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:23.506430 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:23.506525 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:23.510591 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:23.510597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:23.510600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:23.510640 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:23.513205 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:23.514047 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:23.514052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:23.514054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:23.514180 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:23.514629 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:23.514632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:23.514634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:23.514889 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:23.514998 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:23.515869 (+ 871us) server_negotiation.cc:299] Negotiation successful
0812 01:16:23.516058 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
I20260812 01:16:23.517778 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:23.497107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34769 (local address 127.20.151.1:36729)
0812 01:16:23.497989 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:23.498069 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:23.500548 (+ 2479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:23.501263 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:23.501275 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:23.502140 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:23.502978 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:23.503004 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:23.504124 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:23.504130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:23.505900 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:23.505915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:23.506666 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:23.506673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:23.506949 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:23.507668 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:16:23.507696 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:23.510347 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:23.513348 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:23.513357 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:23.513375 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:23.513942 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:23.514277 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:23.514281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:23.514283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:23.514539 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:23.515010 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:23.515019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:23.515466 (+ 447us) client_negotiation.cc:769] Sending connection context
0812 01:16:23.515788 (+ 322us) client_negotiation.cc:240] Negotiation successful
0812 01:16:23.516351 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":337,"threads_started":1}
I20260812 01:16:23.519435 18781 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34769
I20260812 01:16:23.519742 18781 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:23.520483 18781 heartbeater.cc:511] Master 127.20.151.62:34769 requested a full tablet report, sending...
I20260812 01:16:23.522692 18447 ts_manager.cc:194] Registered new tserver with Master: 7640324834eb427ea6fdd78ae6aa63b7 (127.20.151.1:34179)
I20260812 01:16:23.524340 18447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:36729
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:16:23.574263 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34769
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:23.924875 18787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:23.925207 18787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:23.925352 18787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:23.932847 18787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:23.932986 18787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:23.933143 18787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:16:23.943099 18787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:23.944468 18787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:23.945809 18787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:23.958905 18794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:23.959059 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:23.959061 18797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:23.960110 18787 server_base.cc:1034] running on GCE node
I20260812 01:16:23.960743 18787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:23.961675 18787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:23.962885 18787 hybrid_clock.cc:648] HybridClock initialized: now 1786497383962840 us; error 30 us; skew 500 ppm
I20260812 01:16:23.963215 18787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:23 dist-test-slave-21b9 krb5kdc[18372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497383, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:23.966457 18787 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:16:23.967903 18787 webserver.cc:460] Webserver started at http://127.20.151.2:40445/ using document root <none> and password file <none>
I20260812 01:16:23.968649 18787 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:23.968775 18787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:23.969081 18787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:23.972071 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "ac6b93671cd748c29fe724a14cd17d39"
format_stamp: "Formatted at 2026-08-12 01:16:23 on dist-test-slave-21b9"
server_key: "c3f614b10dfb1138e28853d6d736a63f"
server_key_iv: "27ed821c64f6979c5b1e11618145c289"
server_key_version: "encryptionkey@0"
I20260812 01:16:23.972853 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "ac6b93671cd748c29fe724a14cd17d39"
format_stamp: "Formatted at 2026-08-12 01:16:23 on dist-test-slave-21b9"
server_key: "c3f614b10dfb1138e28853d6d736a63f"
server_key_iv: "27ed821c64f6979c5b1e11618145c289"
server_key_version: "encryptionkey@0"
I20260812 01:16:23.978186 18787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260812 01:16:23.981746 18812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:23.983098 18787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 01:16:23.983256 18787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "ac6b93671cd748c29fe724a14cd17d39"
format_stamp: "Formatted at 2026-08-12 01:16:23 on dist-test-slave-21b9"
server_key: "c3f614b10dfb1138e28853d6d736a63f"
server_key_iv: "27ed821c64f6979c5b1e11618145c289"
server_key_version: "encryptionkey@0"
I20260812 01:16:23.983491 18787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:23.997387 18787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:24.001848 18787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:24.002125 18787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:24.003158 18787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:24.004573 18787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:24.004657 18787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:24.004789 18787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:24.004846 18787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:24.042395 18787 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:38359
I20260812 01:16:24.042438 19035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:38359 every 8 connection(s)
I20260812 01:16:24.043807 18787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:16:24.050640 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 18787
I20260812 01:16:24.050846 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:16:24.051435 21084 external_mini_cluster.cc:1428] Setting key e9dc3e9b27d13b12c8a279fcfd1c8c15
Aug 12 01:16:24 dist-test-slave-21b9 krb5kdc[18372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497383, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:24.064735 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.046456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:50657 (local address 127.20.151.62:34769)
0812 01:16:24.046912 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:24.046917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:24.047962 (+ 1045us) server_negotiation.cc:407] Connection header received
0812 01:16:24.049813 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:24.049818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:24.049902 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:24.049998 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:24.052150 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.053029 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.054778 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.055220 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.055358 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.058859 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:24.058864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:24.058867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:24.058900 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:24.061467 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:24.062262 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:24.062266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:24.062269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:24.062409 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:24.062856 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:24.062860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:24.062862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:24.063151 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:24.063321 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:24.064053 (+ 732us) server_negotiation.cc:299] Negotiation successful
0812 01:16:24.064295 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":240,"threads_started":1}
I20260812 01:16:24.065821 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.047023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34769 (local address 127.20.151.2:50657)
0812 01:16:24.047789 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:24.047827 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:24.049549 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:24.050231 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:24.050243 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:24.051018 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:24.051899 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.051928 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.053221 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.053225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.054623 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.054637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.055464 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.055472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.055723 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.056388 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:16:24.056412 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:24.058623 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:24.061647 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:24.061657 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:24.061677 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:24.062138 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:24.062520 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:24.062525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:24.062529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:24.062739 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:24.063304 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:24.063313 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:24.063684 (+ 371us) client_negotiation.cc:769] Sending connection context
0812 01:16:24.063950 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 01:16:24.064539 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":303,"threads_started":1}
I20260812 01:16:24.067430 19038 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34769
I20260812 01:16:24.067751 19038 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:24.068514 19038 heartbeater.cc:511] Master 127.20.151.62:34769 requested a full tablet report, sending...
I20260812 01:16:24.069996 18447 ts_manager.cc:194] Registered new tserver with Master: ac6b93671cd748c29fe724a14cd17d39 (127.20.151.2:38359)
I20260812 01:16:24.070825 18447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:50657
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:16:24.126379 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34769
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:24.475623 19046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:24.475932 19046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:24.476061 19046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:24.483765 19046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:24.483901 19046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:24.484061 19046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:16:24.494314 19046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:24.495599 19046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:24.496985 19046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:24.509999 19060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:24.510098 19055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:24.510483 19063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:24.510711 19046 server_base.cc:1034] running on GCE node
I20260812 01:16:24.511374 19046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:24.512308 19046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:24.513518 19046 hybrid_clock.cc:648] HybridClock initialized: now 1786497384513458 us; error 45 us; skew 500 ppm
I20260812 01:16:24.513863 19046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:24 dist-test-slave-21b9 krb5kdc[18372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497384, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:24.517144 19046 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:16:24.518671 19046 webserver.cc:460] Webserver started at http://127.20.151.3:36657/ using document root <none> and password file <none>
I20260812 01:16:24.519408 19046 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:24.519538 19046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:24.519856 19046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:24.522794 19046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "4870fb1a9fd04a6b8c675bfa6e091373"
format_stamp: "Formatted at 2026-08-12 01:16:24 on dist-test-slave-21b9"
server_key: "2386fa907eb553b0b4e20ce69afd8c40"
server_key_iv: "d9c2f4b5ff7da7300b301c2716d8c6d5"
server_key_version: "encryptionkey@0"
I20260812 01:16:24.523628 19046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "4870fb1a9fd04a6b8c675bfa6e091373"
format_stamp: "Formatted at 2026-08-12 01:16:24 on dist-test-slave-21b9"
server_key: "2386fa907eb553b0b4e20ce69afd8c40"
server_key_iv: "d9c2f4b5ff7da7300b301c2716d8c6d5"
server_key_version: "encryptionkey@0"
I20260812 01:16:24.527487 18781 heartbeater.cc:503] Master 127.20.151.62:34769 was elected leader, sending a full tablet report...
I20260812 01:16:24.528872 19046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 01:16:24.532509 19079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:24.533907 19046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:16:24.534060 19046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "4870fb1a9fd04a6b8c675bfa6e091373"
format_stamp: "Formatted at 2026-08-12 01:16:24 on dist-test-slave-21b9"
server_key: "2386fa907eb553b0b4e20ce69afd8c40"
server_key_iv: "d9c2f4b5ff7da7300b301c2716d8c6d5"
server_key_version: "encryptionkey@0"
I20260812 01:16:24.534260 19046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:24.547451 19046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:24.551767 19046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:24.552043 19046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:24.553031 19046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:24.554409 19046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:24.554483 19046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:24.554597 19046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:24.554653 19046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:24.586664 19046 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:41031
I20260812 01:16:24.586714 19259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:41031 every 8 connection(s)
I20260812 01:16:24.588039 19046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:16:24.591234 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 19046
I20260812 01:16:24.591463 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:16:24.592037 21084 external_mini_cluster.cc:1428] Setting key 09acd0ba549f799a9ec826ccb0d7a66a
Aug 12 01:16:24 dist-test-slave-21b9 krb5kdc[18372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497384, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:24.609253 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.590627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:44037 (local address 127.20.151.62:34769)
0812 01:16:24.591100 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:24.591105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:24.592175 (+ 1070us) server_negotiation.cc:407] Connection header received
0812 01:16:24.594407 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:24.594411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:24.594499 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:24.594585 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:24.596253 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.597066 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.599209 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.599688 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.599811 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.603570 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:24.603575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:24.603578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:24.603645 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:24.606039 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:24.606960 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:24.606965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:24.606967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:24.607097 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:24.607672 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:24.607677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:24.607680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:24.607967 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:24.608095 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:24.608762 (+ 667us) server_negotiation.cc:299] Negotiation successful
0812 01:16:24.608937 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":237,"threads_started":1}
I20260812 01:16:24.610270 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.591173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:34769 (local address 127.20.151.3:44037)
0812 01:16:24.592020 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:24.592058 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:24.594135 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:24.594857 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:24.594868 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:24.595504 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:24.596098 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.596118 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.597249 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.597252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.599055 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.599072 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.600134 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.600142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.600349 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.601034 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:16:24.601058 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:24.603336 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:24.606184 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:24.606194 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:24.606212 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:24.606845 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:24.607231 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:24.607237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:24.607241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:24.607536 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:24.608071 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:24.608078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:24.608436 (+ 358us) client_negotiation.cc:769] Sending connection context
0812 01:16:24.608687 (+ 251us) client_negotiation.cc:240] Negotiation successful
0812 01:16:24.609116 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":415,"threads_started":1}
I20260812 01:16:24.611634 19260 heartbeater.cc:347] Connected to a master server at 127.20.151.62:34769
I20260812 01:16:24.611960 19260 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:24.612733 19260 heartbeater.cc:511] Master 127.20.151.62:34769 requested a full tablet report, sending...
I20260812 01:16:24.614192 18447 ts_manager.cc:194] Registered new tserver with Master: 4870fb1a9fd04a6b8c675bfa6e091373 (127.20.151.3:41031)
I20260812 01:16:24.615008 18447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:44037
I20260812 01:16:24.619902 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:16:24.637288 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.624471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.20.151.62:34769)
0812 01:16:24.624654 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:24.624660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:24.625065 (+ 405us) server_negotiation.cc:407] Connection header received
0812 01:16:24.625244 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:24.625250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:24.625321 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:24.625405 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:24.626935 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.627769 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.629133 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.629555 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.629677 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.632137 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:24.632142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:24.632145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:24.632179 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:24.634650 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:24.635303 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:24.635309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:24.635313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:24.635451 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:24.635806 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:24.635812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:24.635816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:24.636162 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:24.636390 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:24.636754 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 01:16:24.636984 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 01:16:24.641563 18447 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:16:24.648617 18447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:16:24.683846 19286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.662422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.20.151.2:38359)
0812 01:16:24.675333 (+ 12911us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:24.675342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:24.675374 (+ 32us) server_negotiation.cc:407] Connection header received
0812 01:16:24.675497 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:24.675505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:24.675795 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:24.676910 (+ 1115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:24.677160 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.678910 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.681524 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.682529 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.682654 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.682871 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:24.683259 (+ 388us) server_negotiation.cc:299] Negotiation successful
0812 01:16:24.683471 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12714,"thread_start_us":12595,"threads_started":1}
I20260812 01:16:24.683976 19282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.661961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:38359 (local address 127.0.0.1:34884)
0812 01:16:24.674655 (+ 12694us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:24.674674 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:24.674787 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:24.676063 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:24.676072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:24.676106 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:24.676658 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.676677 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.679080 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.679086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.681376 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.681389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.682763 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.682769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.682958 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.683019 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 01:16:24.683180 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 01:16:24.683489 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12563,"mutex_wait_us":87,"thread_start_us":11587,"threads_started":1}
I20260812 01:16:24.685796 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.661574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:41031 (local address 127.0.0.1:49636)
0812 01:16:24.673039 (+ 11465us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:24.673096 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:24.673326 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:24.676492 (+ 3166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:24.676497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:24.676529 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:24.677022 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.677034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.680576 (+ 3542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.680581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.682707 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.682722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.685117 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.685124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.685294 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.685329 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 01:16:24.685399 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 01:16:24.685507 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11114,"mutex_wait_us":3,"thread_start_us":12796,"threads_started":1}
I20260812 01:16:24.686841 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.661707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.20.151.3:41031)
0812 01:16:24.675836 (+ 14129us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:24.675844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:24.675878 (+ 34us) server_negotiation.cc:407] Connection header received
0812 01:16:24.675984 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:24.675991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:24.676242 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:24.677285 (+ 1043us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:24.677597 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.680422 (+ 2825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.682906 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.684873 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.686140 (+ 1267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.686337 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:24.686403 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 01:16:24.686547 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13971,"thread_start_us":13813,"threads_started":1}
I20260812 01:16:24.687811 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.675158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.20.151.1:34179)
0812 01:16:24.679389 (+ 4231us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:24.679398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:24.679879 (+ 481us) server_negotiation.cc:407] Connection header received
0812 01:16:24.680069 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:24.680077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:24.680327 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:24.680545 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:24.681299 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.682829 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.685191 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.686656 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.686803 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.687017 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:24.687320 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 01:16:24.687480 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4013,"thread_start_us":3839,"threads_started":1}
I20260812 01:16:24.687896 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.675887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34179 (local address 127.0.0.1:56410)
0812 01:16:24.679744 (+ 3857us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:24.679761 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:24.679899 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:24.680554 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:24.680559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:24.680586 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:24.681116 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.681132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.682962 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.682967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.685000 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.685014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.686891 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.686898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.687070 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.687102 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:16:24.687199 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 01:16:24.687426 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3735,"mutex_wait_us":19,"thread_start_us":3574,"threads_started":1}
I20260812 01:16:24.688601 18912 tablet_service.cc:1467] Processing CreateTablet for tablet 57d18de0bacf4ded85c97519adcccae8 (DEFAULT_TABLE table=test-table [id=ecfad15b41f64735a0c0671ee9e98f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:24.690025 18912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d18de0bacf4ded85c97519adcccae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:24.690536 19153 tablet_service.cc:1467] Processing CreateTablet for tablet 57d18de0bacf4ded85c97519adcccae8 (DEFAULT_TABLE table=test-table [id=ecfad15b41f64735a0c0671ee9e98f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:24.692065 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d18de0bacf4ded85c97519adcccae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:24.692852 18671 tablet_service.cc:1467] Processing CreateTablet for tablet 57d18de0bacf4ded85c97519adcccae8 (DEFAULT_TABLE table=test-table [id=ecfad15b41f64735a0c0671ee9e98f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:24.694268 18671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d18de0bacf4ded85c97519adcccae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:24.702752 19290 tablet_bootstrap.cc:492] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: Bootstrap starting.
I20260812 01:16:24.706823 19291 tablet_bootstrap.cc:492] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: Bootstrap starting.
I20260812 01:16:24.706985 19290 tablet_bootstrap.cc:654] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:24.708355 19290 log.cc:824] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:24.708546 19293 tablet_bootstrap.cc:492] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: Bootstrap starting.
I20260812 01:16:24.710425 19291 tablet_bootstrap.cc:654] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:24.711735 19291 log.cc:824] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:24.711833 19293 tablet_bootstrap.cc:654] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:24.713105 19290 tablet_bootstrap.cc:492] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: No bootstrap required, opened a new log
I20260812 01:16:24.713094 19293 log.cc:824] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:24.713397 19290 ts_tablet_manager.cc:1397] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260812 01:16:24.715045 19291 tablet_bootstrap.cc:492] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: No bootstrap required, opened a new log
I20260812 01:16:24.715351 19291 ts_tablet_manager.cc:1397] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260812 01:16:24.717234 19293 tablet_bootstrap.cc:492] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: No bootstrap required, opened a new log
I20260812 01:16:24.717483 19293 ts_tablet_manager.cc:1397] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 01:16:24.724135 19290 raft_consensus.cc:348] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41031 } } peers { permanent_uuid: "7640324834eb427ea6fdd78ae6aa63b7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34179 } } peers { permanent_uuid: "ac6b93671cd748c29fe724a14cd17d39" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 38359 } }
I20260812 01:16:24.724828 19290 raft_consensus.cc:374] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:24.724634 19293 raft_consensus.cc:348] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41031 } } peers { permanent_uuid: "7640324834eb427ea6fdd78ae6aa63b7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34179 } } peers { permanent_uuid: "ac6b93671cd748c29fe724a14cd17d39" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 38359 } }
I20260812 01:16:24.725003 19290 raft_consensus.cc:729] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6b93671cd748c29fe724a14cd17d39, State: Initialized, Role: FOLLOWER
I20260812 01:16:24.725185 19293 raft_consensus.cc:374] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:24.725366 19293 raft_consensus.cc:729] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7640324834eb427ea6fdd78ae6aa63b7, State: Initialized, Role: FOLLOWER
I20260812 01:16:24.725843 19290 consensus_queue.cc:260] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41031 } } peers { permanent_uuid: "7640324834eb427ea6fdd78ae6aa63b7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34179 } } peers { permanent_uuid: "ac6b93671cd748c29fe724a14cd17d39" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 38359 } }
I20260812 01:16:24.725920 19291 raft_consensus.cc:348] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41031 } } peers { permanent_uuid: "7640324834eb427ea6fdd78ae6aa63b7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34179 } } peers { permanent_uuid: "ac6b93671cd748c29fe724a14cd17d39" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 38359 } }
I20260812 01:16:24.726085 19293 consensus_queue.cc:260] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41031 } } peers { permanent_uuid: "7640324834eb427ea6fdd78ae6aa63b7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34179 } } peers { permanent_uuid: "ac6b93671cd748c29fe724a14cd17d39" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 38359 } }
I20260812 01:16:24.726368 19291 raft_consensus.cc:374] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:24.726476 19291 raft_consensus.cc:729] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4870fb1a9fd04a6b8c675bfa6e091373, State: Initialized, Role: FOLLOWER
I20260812 01:16:24.727232 19291 consensus_queue.cc:260] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41031 } } peers { permanent_uuid: "7640324834eb427ea6fdd78ae6aa63b7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34179 } } peers { permanent_uuid: "ac6b93671cd748c29fe724a14cd17d39" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 38359 } }
I20260812 01:16:24.727856 19293 ts_tablet_manager.cc:1428] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260812 01:16:24.728914 19260 heartbeater.cc:503] Master 127.20.151.62:34769 was elected leader, sending a full tablet report...
I20260812 01:16:24.729525 19291 ts_tablet_manager.cc:1428] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260812 01:16:24.730968 19290 ts_tablet_manager.cc:1428] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260812 01:16:24.731369 19038 heartbeater.cc:503] Master 127.20.151.62:34769 was elected leader, sending a full tablet report...
W20260812 01:16:24.746459 18782 tablet.cc:2369] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:16:24.795964 19039 tablet.cc:2369] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:16:24.823366 19299 raft_consensus.cc:482] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:16:24.823707 19299 raft_consensus.cc:504] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41031 } } peers { permanent_uuid: "7640324834eb427ea6fdd78ae6aa63b7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34179 } } peers { permanent_uuid: "ac6b93671cd748c29fe724a14cd17d39" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 38359 } }
I20260812 01:16:24.825477 19299 leader_election.cc:288] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4870fb1a9fd04a6b8c675bfa6e091373 (127.20.151.3:41031), 7640324834eb427ea6fdd78ae6aa63b7 (127.20.151.1:34179)
I20260812 01:16:24.831905 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.825921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:41365 (local address 127.20.151.3:41031)
0812 01:16:24.826106 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:24.826111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:24.826500 (+ 389us) server_negotiation.cc:407] Connection header received
0812 01:16:24.826793 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:24.826805 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:24.826894 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:24.826960 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:24.827655 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.828405 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.830209 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.830973 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.831125 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.831205 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:24.831527 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 01:16:24.831661 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:16:24.832002 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.825710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:41031 (local address 127.20.151.2:41365)
0812 01:16:24.826391 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:24.826410 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:24.826563 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:24.827150 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:24.827154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:24.827184 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:24.827515 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.827524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.828534 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.828538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.830049 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.830062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.831177 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.831183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.831374 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.831399 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:16:24.831492 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 01:16:24.831640 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":328,"threads_started":1}
I20260812 01:16:24.832216 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.826655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:52375 (local address 127.20.151.1:34179)
0812 01:16:24.826914 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:24.826920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:24.827184 (+ 264us) server_negotiation.cc:407] Connection header received
0812 01:16:24.827394 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:24.827400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:24.827462 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:24.827536 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:24.828075 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.828765 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.830485 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.831296 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.831440 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.831516 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:24.831842 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 01:16:24.831972 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 01:16:24.832326 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.826485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34179 (local address 127.20.151.2:52375)
0812 01:16:24.827081 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:24.827095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:24.827229 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:24.827582 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:24.827586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:24.827610 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:24.827926 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.827934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.828902 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.828906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.830331 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:24.830343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.831505 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.831511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:24.831697 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.831721 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 01:16:24.831808 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 01:16:24.831928 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":1,"thread_start_us":298,"threads_started":1}
I20260812 01:16:24.833185 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d18de0bacf4ded85c97519adcccae8" candidate_uuid: "ac6b93671cd748c29fe724a14cd17d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" is_pre_election: true
I20260812 01:16:24.833267 18707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d18de0bacf4ded85c97519adcccae8" candidate_uuid: "ac6b93671cd748c29fe724a14cd17d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7640324834eb427ea6fdd78ae6aa63b7" is_pre_election: true
I20260812 01:16:24.833714 19194 raft_consensus.cc:2454] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6b93671cd748c29fe724a14cd17d39 in term 0.
I20260812 01:16:24.833736 18707 raft_consensus.cc:2454] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6b93671cd748c29fe724a14cd17d39 in term 0.
I20260812 01:16:24.834380 18818 leader_election.cc:302] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7640324834eb427ea6fdd78ae6aa63b7, ac6b93671cd748c29fe724a14cd17d39; no voters:
I20260812 01:16:24.834756 19299 raft_consensus.cc:2781] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:16:24.834933 19299 raft_consensus.cc:482] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:16:24.835045 19299 raft_consensus.cc:3037] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:24.838269 19299 raft_consensus.cc:504] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41031 } } peers { permanent_uuid: "7640324834eb427ea6fdd78ae6aa63b7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34179 } } peers { permanent_uuid: "ac6b93671cd748c29fe724a14cd17d39" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 38359 } }
I20260812 01:16:24.838938 19299 leader_election.cc:288] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [CANDIDATE]: Term 1 election: Requested vote from peers 4870fb1a9fd04a6b8c675bfa6e091373 (127.20.151.3:41031), 7640324834eb427ea6fdd78ae6aa63b7 (127.20.151.1:34179)
I20260812 01:16:24.839427 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d18de0bacf4ded85c97519adcccae8" candidate_uuid: "ac6b93671cd748c29fe724a14cd17d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4870fb1a9fd04a6b8c675bfa6e091373"
I20260812 01:16:24.839483 18707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d18de0bacf4ded85c97519adcccae8" candidate_uuid: "ac6b93671cd748c29fe724a14cd17d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7640324834eb427ea6fdd78ae6aa63b7"
I20260812 01:16:24.839663 19194 raft_consensus.cc:3037] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373 [term 0 FOLLOWER]: Advancing to term 1
W20260812 01:16:24.839670 19262 tablet.cc:2369] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:16:24.839712 18707 raft_consensus.cc:3037] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:24.842098 18707 raft_consensus.cc:2454] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6b93671cd748c29fe724a14cd17d39 in term 1.
I20260812 01:16:24.842489 18818 leader_election.cc:302] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7640324834eb427ea6fdd78ae6aa63b7, ac6b93671cd748c29fe724a14cd17d39; no voters:
I20260812 01:16:24.842756 19299 raft_consensus.cc:2781] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:24.842895 19194 raft_consensus.cc:2454] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6b93671cd748c29fe724a14cd17d39 in term 1.
I20260812 01:16:24.843312 19299 raft_consensus.cc:686] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 1 LEADER]: Becoming Leader. State: Replica: ac6b93671cd748c29fe724a14cd17d39, State: Running, Role: LEADER
I20260812 01:16:24.844132 19299 consensus_queue.cc:237] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41031 } } peers { permanent_uuid: "7640324834eb427ea6fdd78ae6aa63b7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34179 } } peers { permanent_uuid: "ac6b93671cd748c29fe724a14cd17d39" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 38359 } }
I20260812 01:16:24.848609 18445 catalog_manager.cc:5495] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6b93671cd748c29fe724a14cd17d39 (127.20.151.2). New cstate: current_term: 1 leader_uuid: "ac6b93671cd748c29fe724a14cd17d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7640324834eb427ea6fdd78ae6aa63b7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac6b93671cd748c29fe724a14cd17d39" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 38359 } health_report { overall_health: HEALTHY } } }
I20260812 01:16:24.864423 19286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.859642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.20.151.2:38359)
0812 01:16:24.859825 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:24.859829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:24.859847 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:16:24.859932 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:24.859935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:24.860009 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:24.860143 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:16:24.860439 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.861195 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.862530 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.862929 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.863019 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.863453 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:16:24.863952 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:16:24.864079 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:24.864116 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 01:16:24.864183 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 01:16:24.872380 18707 raft_consensus.cc:1261] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7 [term 1 FOLLOWER]: Refusing update from remote peer ac6b93671cd748c29fe724a14cd17d39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:24.872512 19194 raft_consensus.cc:1261] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373 [term 1 FOLLOWER]: Refusing update from remote peer ac6b93671cd748c29fe724a14cd17d39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:24.873359 19308 consensus_queue.cc:1035] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7640324834eb427ea6fdd78ae6aa63b7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:16:24.873744 19299 consensus_queue.cc:1035] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4870fb1a9fd04a6b8c675bfa6e091373" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 41031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:16:24.884754 19313 mvcc.cc:205] Tried to move back new op lower bound from 7317493288427810816 to 7317493288325427200. Current Snapshot: MvccSnapshot[applied={T|T < 7317493288427810816}]
I20260812 01:16:24.889415 19316 mvcc.cc:205] Tried to move back new op lower bound from 7317493288427810816 to 7317493288325427200. Current Snapshot: MvccSnapshot[applied={T|T < 7317493288427810816}]
I20260812 01:16:24.898733 18447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:24.899051 18447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:24.902947 18447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 663443db6f8c41da94af1e18599ace43: Sending DeleteTablet for 3 replicas of tablet 57d18de0bacf4ded85c97519adcccae8
I20260812 01:16:24.904230 18912 tablet_service.cc:1514] Processing DeleteTablet for tablet 57d18de0bacf4ded85c97519adcccae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:24 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:34884
I20260812 01:16:24.904291 19153 tablet_service.cc:1514] Processing DeleteTablet for tablet 57d18de0bacf4ded85c97519adcccae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:24 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:49636
I20260812 01:16:24.905258 18671 tablet_service.cc:1514] Processing DeleteTablet for tablet 57d18de0bacf4ded85c97519adcccae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:24 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:56410
I20260812 01:16:24.905943 19332 tablet_replica.cc:330] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: stopping tablet replica
I20260812 01:16:24.906327 19332 raft_consensus.cc:2229] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:16:24.906894 19332 raft_consensus.cc:2258] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:24.908070 19334 tablet_replica.cc:330] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: stopping tablet replica
I20260812 01:16:24.908499 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 18559
I20260812 01:16:24.908468 19334 raft_consensus.cc:2229] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:24.909000 19334 raft_consensus.cc:2258] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:24.909799 19333 tablet_replica.cc:330] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: stopping tablet replica
I20260812 01:16:24.910179 19333 raft_consensus.cc:2229] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:24.910691 19333 raft_consensus.cc:2258] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:24.912480 19334 ts_tablet_manager.cc:1905] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:24.912964 19333 ts_tablet_manager.cc:1905] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:24.913261 19332 ts_tablet_manager.cc:1905] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:24.919935 19334 ts_tablet_manager.cc:1918] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:24.920187 19334 log.cc:1196] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/57d18de0bacf4ded85c97519adcccae8
I20260812 01:16:24.920785 19334 ts_tablet_manager.cc:1939] T 57d18de0bacf4ded85c97519adcccae8 P 7640324834eb427ea6fdd78ae6aa63b7: Deleting consensus metadata
I20260812 01:16:24.921491 19332 ts_tablet_manager.cc:1918] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 12 01:16:24 dist-test-slave-21b9 krb5kdc[18372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:16:24.921757 19332 log.cc:1196] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/57d18de0bacf4ded85c97519adcccae8
I20260812 01:16:24.922483 19332 ts_tablet_manager.cc:1939] T 57d18de0bacf4ded85c97519adcccae8 P ac6b93671cd748c29fe724a14cd17d39: Deleting consensus metadata
I20260812 01:16:24.922948 18428 catalog_manager.cc:4841] TS 7640324834eb427ea6fdd78ae6aa63b7 (127.20.151.1:34179): tablet 57d18de0bacf4ded85c97519adcccae8 (table test-table [id=ecfad15b41f64735a0c0671ee9e98f40]) successfully deleted
I20260812 01:16:24.924216 18424 catalog_manager.cc:4841] TS ac6b93671cd748c29fe724a14cd17d39 (127.20.151.2:38359): tablet 57d18de0bacf4ded85c97519adcccae8 (table test-table [id=ecfad15b41f64735a0c0671ee9e98f40]) successfully deleted
I20260812 01:16:24.928869 19333 ts_tablet_manager.cc:1918] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:24.929060 19333 log.cc:1196] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/57d18de0bacf4ded85c97519adcccae8
I20260812 01:16:24.929666 19333 ts_tablet_manager.cc:1939] T 57d18de0bacf4ded85c97519adcccae8 P 4870fb1a9fd04a6b8c675bfa6e091373: Deleting consensus metadata
I20260812 01:16:24.929718 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:24.909760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56426 (local address 127.20.151.1:34179)
0812 01:16:24.909930 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:24.909936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:24.910586 (+ 650us) server_negotiation.cc:407] Connection header received
0812 01:16:24.910863 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:24.910870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:24.910951 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:24.911032 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:24.913417 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.914646 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.916665 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:24.917326 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:24.917430 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:24.923184 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:24.923200 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:24.923219 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:24.923289 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:24.926540 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:24.927392 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:24.927400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:24.927402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:24.927544 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:24.928066 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:24.928072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:24.928076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:24.928469 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:24.928695 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:24.929241 (+ 546us) server_negotiation.cc:299] Negotiation successful
0812 01:16:24.929430 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 01:16:24.931030 18770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:24.931620 18428 catalog_manager.cc:4841] TS 4870fb1a9fd04a6b8c675bfa6e091373 (127.20.151.3:41031): tablet 57d18de0bacf4ded85c97519adcccae8 (table test-table [id=ecfad15b41f64735a0c0671ee9e98f40]) successfully deleted
I20260812 01:16:25.023864 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 18559
I20260812 01:16:25.059733 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 18787
Aug 12 01:16:25 dist-test-slave-21b9 krb5kdc[18372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:16:25.077703 19286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:25.060536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34906 (local address 127.20.151.2:38359)
0812 01:16:25.060774 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:25.060780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:25.060854 (+ 74us) server_negotiation.cc:407] Connection header received
0812 01:16:25.061054 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:25.061060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:25.061135 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:25.061244 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:25.062864 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:25.063768 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:25.065702 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:25.066178 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:25.066271 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:25.071402 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:25.071417 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:25.071434 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:25.071474 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:25.074682 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:25.075457 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:25.075465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:25.075468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:25.075631 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:25.076023 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:25.076026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:25.076029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:25.076421 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:25.076788 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:25.077027 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 01:16:25.077256 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260812 01:16:25.078347 19027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:25.193681 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 18787
I20260812 01:16:25.221432 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 19046
Aug 12 01:16:25 dist-test-slave-21b9 krb5kdc[18372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:16:25.236356 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:25.222144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.20.151.3:41031)
0812 01:16:25.222328 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:25.222333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:25.222370 (+ 37us) server_negotiation.cc:407] Connection header received
0812 01:16:25.222535 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:25.222539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:25.222625 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:25.222706 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:25.224086 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:25.224819 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:25.226064 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:25.226441 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:25.226528 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:25.231009 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:25.231025 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:25.231042 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:25.231076 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:25.233809 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:25.234493 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:25.234499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:25.234502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:25.234655 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:25.234969 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:25.234971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:25.234974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:25.235373 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:25.235592 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:25.235859 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 01:16:25.236029 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 01:16:25.237058 19254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:25.327306 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 19046
I20260812 01:16:25.353375 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 18397
I20260812 01:16:25.354789 18508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:25.466822 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 18397
2026-08-12T01:16:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4752 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T01:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:25Z Disabled control of system clock
I20260812 01:16:25.577494 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:40211
--webserver_interface=127.20.151.62
--webserver_port=0
--builtin_ntp_servers=127.20.151.20:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:40211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:16:25.929445 19352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:25.929785 19352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:25.929917 19352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:25.937307 19352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:25.937451 19352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:25.937521 19352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:25.937613 19352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:25.937676 19352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:25.947835 19352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:40211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:40211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:25.949254 19352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:25.951117 19352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:25.962589 19362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:25.962704 19363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:25.962677 19365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:25.964058 19352 server_base.cc:1034] running on GCE node
I20260812 01:16:25.964866 19352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:25.966364 19352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:25.967557 19352 hybrid_clock.cc:648] HybridClock initialized: now 1786497385967506 us; error 54 us; skew 500 ppm
I20260812 01:16:25.967892 19352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:25.970189 19352 webserver.cc:460] Webserver started at http://127.20.151.62:37789/ using document root <none> and password file <none>
I20260812 01:16:25.970870 19352 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:25.970975 19352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:25.971333 19352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:25.974254 19352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "1bb89895d49c4bfaafce92971e0d303d"
format_stamp: "Formatted at 2026-08-12 01:16:25 on dist-test-slave-21b9"
server_key: "1c4401d9f4d3a5b33b30cdb10473b931"
server_key_iv: "6f377454346e8379e308a50bd3acee8e"
server_key_version: "encryptionkey@0"
I20260812 01:16:25.975044 19352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "1bb89895d49c4bfaafce92971e0d303d"
format_stamp: "Formatted at 2026-08-12 01:16:25 on dist-test-slave-21b9"
server_key: "1c4401d9f4d3a5b33b30cdb10473b931"
server_key_iv: "6f377454346e8379e308a50bd3acee8e"
server_key_version: "encryptionkey@0"
I20260812 01:16:25.980206 19352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260812 01:16:25.983636 19379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:25.984901 19352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 01:16:25.985074 19352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "1bb89895d49c4bfaafce92971e0d303d"
format_stamp: "Formatted at 2026-08-12 01:16:25 on dist-test-slave-21b9"
server_key: "1c4401d9f4d3a5b33b30cdb10473b931"
server_key_iv: "6f377454346e8379e308a50bd3acee8e"
server_key_version: "encryptionkey@0"
I20260812 01:16:25.985291 19352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:26.000299 19352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:26.001165 19352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:26.001447 19352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:26.022894 19352 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:40211
I20260812 01:16:26.022900 19470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:40211 every 8 connection(s)
I20260812 01:16:26.024572 19352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:26.028613 19471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:26.032374 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 19352
I20260812 01:16:26.032557 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:26.033008 21084 external_mini_cluster.cc:1428] Setting key 366e2bf3def98f99111ae79b2e59931b
I20260812 01:16:26.043483 19471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d: Bootstrap starting.
I20260812 01:16:26.047676 19471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:26.049309 19471 log.cc:824] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:26.049310 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:26.035129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.20.151.62:40211)
0812 01:16:26.036279 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:26.036298 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:26.036353 (+ 55us) server_negotiation.cc:407] Connection header received
0812 01:16:26.038494 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:26.038529 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:26.039224 (+ 695us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:26.039801 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:26.040410 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:26.042390 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:26.043927 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:26.044977 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:26.045167 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:26.046268 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:26.046280 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:26.046299 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:26.047066 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:26.047197 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:26.047226 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:26.047373 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:26.047578 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 01:16:26.047933 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":376,"threads_started":1}
I20260812 01:16:26.053124 19471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d: No bootstrap required, opened a new log
I20260812 01:16:26.061131 19471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb89895d49c4bfaafce92971e0d303d" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40211 } }
I20260812 01:16:26.061527 19471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:26.061607 19471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb89895d49c4bfaafce92971e0d303d, State: Initialized, Role: FOLLOWER
I20260812 01:16:26.062171 19471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb89895d49c4bfaafce92971e0d303d" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40211 } }
I20260812 01:16:26.062397 19471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:26.062503 19471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:26.062641 19471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:26.065276 19471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb89895d49c4bfaafce92971e0d303d" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40211 } }
I20260812 01:16:26.065762 19471 leader_election.cc:302] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb89895d49c4bfaafce92971e0d303d; no voters:
I20260812 01:16:26.066332 19471 leader_election.cc:288] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:26.066454 19477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:26.067230 19477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [term 1 LEADER]: Becoming Leader. State: Replica: 1bb89895d49c4bfaafce92971e0d303d, State: Running, Role: LEADER
I20260812 01:16:26.067876 19477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb89895d49c4bfaafce92971e0d303d" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40211 } }
I20260812 01:16:26.068578 19471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:26.070681 19480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bb89895d49c4bfaafce92971e0d303d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb89895d49c4bfaafce92971e0d303d" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40211 } } }
I20260812 01:16:26.071154 19480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:26.071617 19481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bb89895d49c4bfaafce92971e0d303d. Latest consensus state: current_term: 1 leader_uuid: "1bb89895d49c4bfaafce92971e0d303d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb89895d49c4bfaafce92971e0d303d" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 40211 } } }
I20260812 01:16:26.071990 19481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:26.076423 19491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:26.081215 19491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:26.090358 19491 catalog_manager.cc:1347] Generated new cluster ID: 2b97933d07b642d3ac3bf3723ecd45e2
I20260812 01:16:26.090493 19491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:26.109582 19491 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:26.110888 19491 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:26.118880 19491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d: Generated new TSK 0
I20260812 01:16:26.119982 19491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:16:26.136229 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40211
--builtin_ntp_servers=127.20.151.20:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:16:26.510666 19508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:26.510986 19508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:26.511125 19508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:26.518640 19508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:26.518765 19508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:26.518915 19508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:16:26.528411 19508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:26.529789 19508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:26.531715 19508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:26.545042 19513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:26.545049 19516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:26.545109 19514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:26.546247 19508 server_base.cc:1034] running on GCE node
I20260812 01:16:26.546878 19508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:26.547966 19508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:26.549243 19508 hybrid_clock.cc:648] HybridClock initialized: now 1786497386549144 us; error 82 us; skew 500 ppm
I20260812 01:16:26.549566 19508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:26.551968 19508 webserver.cc:460] Webserver started at http://127.20.151.1:43965/ using document root <none> and password file <none>
I20260812 01:16:26.552655 19508 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:26.552748 19508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:26.553025 19508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:26.556012 19508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "3d71fd633576459cad4c692fc7c5f3f1"
format_stamp: "Formatted at 2026-08-12 01:16:26 on dist-test-slave-21b9"
server_key: "a00e88b98b79b4903a44be6a32f91900"
server_key_iv: "7aa9a6ff96c61b2a2f35f7884c655a50"
server_key_version: "encryptionkey@0"
I20260812 01:16:26.556761 19508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "3d71fd633576459cad4c692fc7c5f3f1"
format_stamp: "Formatted at 2026-08-12 01:16:26 on dist-test-slave-21b9"
server_key: "a00e88b98b79b4903a44be6a32f91900"
server_key_iv: "7aa9a6ff96c61b2a2f35f7884c655a50"
server_key_version: "encryptionkey@0"
I20260812 01:16:26.562017 19508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 01:16:26.565614 19527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:26.566903 19508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 01:16:26.567076 19508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "3d71fd633576459cad4c692fc7c5f3f1"
format_stamp: "Formatted at 2026-08-12 01:16:26 on dist-test-slave-21b9"
server_key: "a00e88b98b79b4903a44be6a32f91900"
server_key_iv: "7aa9a6ff96c61b2a2f35f7884c655a50"
server_key_version: "encryptionkey@0"
I20260812 01:16:26.567339 19508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:26.586802 19508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:26.587733 19508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:26.588068 19508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:26.589129 19508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:26.590703 19508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:26.590807 19508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:26.590915 19508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:26.591002 19508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:26.624673 19508 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:34857
I20260812 01:16:26.624720 19712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:34857 every 8 connection(s)
I20260812 01:16:26.625988 19508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:16:26.632500 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 19508
I20260812 01:16:26.632680 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:16:26.633126 21084 external_mini_cluster.cc:1428] Setting key 8a24a293a1539eba106e944018d3332a
I20260812 01:16:26.635875 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40211
--builtin_ntp_servers=127.20.151.20:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:16:26.638764 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:26.628580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:54333 (local address 127.20.151.62:40211)
0812 01:16:26.629028 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:26.629034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:26.630278 (+ 1244us) server_negotiation.cc:407] Connection header received
0812 01:16:26.632184 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:26.632190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:26.632274 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:26.632374 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:26.633536 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:26.634258 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:26.635899 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:26.636370 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:26.636481 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:26.637894 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:26.637900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:26.637903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:26.637921 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:26.637961 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:26.637973 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:26.638048 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:26.638291 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 01:16:26.638393 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
I20260812 01:16:26.639807 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:26.629190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40211 (local address 127.20.151.1:54333)
0812 01:16:26.630107 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:26.630143 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:26.631894 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:26.632622 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:26.632631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:26.632738 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:26.633369 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:26.633388 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:26.634427 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:26.634431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:26.635754 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:26.635767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:26.636609 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:26.636618 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:26.636842 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:26.637559 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:16:26.637562 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:26.637607 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:16:26.637700 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:26.638062 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:26.638074 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:26.638094 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 01:16:26.638256 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 01:16:26.638681 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":292,"threads_started":1}
I20260812 01:16:26.641319 19713 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40211
I20260812 01:16:26.641672 19713 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:26.642624 19713 heartbeater.cc:511] Master 127.20.151.62:40211 requested a full tablet report, sending...
I20260812 01:16:26.644873 19410 ts_manager.cc:194] Registered new tserver with Master: 3d71fd633576459cad4c692fc7c5f3f1 (127.20.151.1:34857)
I20260812 01:16:26.646368 19410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.1:54333
W20260812 01:16:26.988293 19717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:26.988623 19717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:26.988757 19717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:26.996338 19717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:26.996450 19717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:26.996599 19717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:16:27.005887 19717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:27.007210 19717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:27.009155 19717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:27.022238 19732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:27.022372 19731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:27.022202 19734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:27.023133 19717 server_base.cc:1034] running on GCE node
I20260812 01:16:27.023782 19717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:27.024757 19717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:27.025976 19717 hybrid_clock.cc:648] HybridClock initialized: now 1786497387025937 us; error 55 us; skew 500 ppm
I20260812 01:16:27.026294 19717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:27.028687 19717 webserver.cc:460] Webserver started at http://127.20.151.2:37381/ using document root <none> and password file <none>
I20260812 01:16:27.029448 19717 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:27.029565 19717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:27.029875 19717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:27.032924 19717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "e696d4eb4e6742a3ba4034f0771e08c9"
format_stamp: "Formatted at 2026-08-12 01:16:27 on dist-test-slave-21b9"
server_key: "b372332d5ee8facce218fb1acd94b2ba"
server_key_iv: "c3a3c633a96cd317bbacb148294b7405"
server_key_version: "encryptionkey@0"
I20260812 01:16:27.033761 19717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "e696d4eb4e6742a3ba4034f0771e08c9"
format_stamp: "Formatted at 2026-08-12 01:16:27 on dist-test-slave-21b9"
server_key: "b372332d5ee8facce218fb1acd94b2ba"
server_key_iv: "c3a3c633a96cd317bbacb148294b7405"
server_key_version: "encryptionkey@0"
I20260812 01:16:27.039198 19717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:16:27.042898 19744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:27.044370 19717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:16:27.044600 19717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "e696d4eb4e6742a3ba4034f0771e08c9"
format_stamp: "Formatted at 2026-08-12 01:16:27 on dist-test-slave-21b9"
server_key: "b372332d5ee8facce218fb1acd94b2ba"
server_key_iv: "c3a3c633a96cd317bbacb148294b7405"
server_key_version: "encryptionkey@0"
I20260812 01:16:27.044832 19717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:27.104629 19717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:27.105576 19717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:27.105888 19717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:27.107033 19717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:27.108681 19717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:27.108788 19717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:27.108904 19717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:27.108986 19717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:27.140487 19717 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:46441
I20260812 01:16:27.140517 19953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:46441 every 8 connection(s)
I20260812 01:16:27.141875 19717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:16:27.143100 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 19717
I20260812 01:16:27.143340 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:16:27.143944 21084 external_mini_cluster.cc:1428] Setting key 9958190774c2d0e6c832d130e7be9890
I20260812 01:16:27.147215 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40211
--builtin_ntp_servers=127.20.151.20:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:16:27.155120 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:27.145149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:40073 (local address 127.20.151.62:40211)
0812 01:16:27.145636 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:27.145642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:27.146741 (+ 1099us) server_negotiation.cc:407] Connection header received
0812 01:16:27.148664 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:27.148669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:27.148755 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:27.148850 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:27.150046 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.150789 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.152393 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.152809 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.152922 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:27.154262 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:27.154267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:27.154270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:27.154289 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:27.154330 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:27.154343 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:27.154418 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:27.154648 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 01:16:27.154752 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":227,"threads_started":1}
I20260812 01:16:27.156194 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:27.145584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40211 (local address 127.20.151.2:40073)
0812 01:16:27.146581 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:27.146634 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:27.148394 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:27.149115 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:27.149123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:27.149236 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:27.149873 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:27.149898 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.150948 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.150952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:27.152251 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:27.152265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.153035 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.153043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:27.153263 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:27.153934 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:16:27.153936 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:27.153985 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:16:27.154074 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:27.154432 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:27.154444 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:27.154452 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 01:16:27.154631 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 01:16:27.155034 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":325,"threads_started":1}
I20260812 01:16:27.157655 19954 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40211
I20260812 01:16:27.157994 19954 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:27.158918 19954 heartbeater.cc:511] Master 127.20.151.62:40211 requested a full tablet report, sending...
I20260812 01:16:27.160518 19410 ts_manager.cc:194] Registered new tserver with Master: e696d4eb4e6742a3ba4034f0771e08c9 (127.20.151.2:46441)
I20260812 01:16:27.161453 19410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.2:40073
W20260812 01:16:27.496755 19963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:27.497094 19963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:27.497193 19963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:27.504359 19963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:27.504477 19963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:27.504588 19963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:16:27.513805 19963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:40211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:27.515045 19963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:27.516800 19963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:27.529675 19974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:27.529675 19979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:27.529698 19975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:27.530603 19963 server_base.cc:1034] running on GCE node
I20260812 01:16:27.531204 19963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:27.532219 19963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:27.533468 19963 hybrid_clock.cc:648] HybridClock initialized: now 1786497387533388 us; error 44 us; skew 500 ppm
I20260812 01:16:27.533808 19963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:27.536324 19963 webserver.cc:460] Webserver started at http://127.20.151.3:46049/ using document root <none> and password file <none>
I20260812 01:16:27.537043 19963 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:27.537161 19963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:27.537485 19963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:27.540594 19963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "248590f8437b4debbdfb91f4f24235d9"
format_stamp: "Formatted at 2026-08-12 01:16:27 on dist-test-slave-21b9"
server_key: "5cbaa8d7ecba6aea37d118454b44e1a0"
server_key_iv: "0fbabf9f3adb7a76d3aa6825918c75f2"
server_key_version: "encryptionkey@0"
I20260812 01:16:27.541378 19963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "248590f8437b4debbdfb91f4f24235d9"
format_stamp: "Formatted at 2026-08-12 01:16:27 on dist-test-slave-21b9"
server_key: "5cbaa8d7ecba6aea37d118454b44e1a0"
server_key_iv: "0fbabf9f3adb7a76d3aa6825918c75f2"
server_key_version: "encryptionkey@0"
I20260812 01:16:27.546517 19963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260812 01:16:27.550379 19994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:27.551782 19963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:16:27.551940 19963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "248590f8437b4debbdfb91f4f24235d9"
format_stamp: "Formatted at 2026-08-12 01:16:27 on dist-test-slave-21b9"
server_key: "5cbaa8d7ecba6aea37d118454b44e1a0"
server_key_iv: "0fbabf9f3adb7a76d3aa6825918c75f2"
server_key_version: "encryptionkey@0"
I20260812 01:16:27.552155 19963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:27.573590 19963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:27.574437 19963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:27.574728 19963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:27.575753 19963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:27.577292 19963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:27.577390 19963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:27.577505 19963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:27.577574 19963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:27.610189 19963 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:33035
I20260812 01:16:27.610268 20197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:33035 every 8 connection(s)
I20260812 01:16:27.611596 19963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:16:27.613481 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 19963
I20260812 01:16:27.613670 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:16:27.614136 21084 external_mini_cluster.cc:1428] Setting key 769082fdc69040c01dfb326f616ecb8a
I20260812 01:16:27.624475 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:27.614455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:46559 (local address 127.20.151.62:40211)
0812 01:16:27.615528 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:27.615532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:27.616254 (+ 722us) server_negotiation.cc:407] Connection header received
0812 01:16:27.618174 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:27.618179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:27.618268 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:27.618356 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:27.619481 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.620239 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.621800 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.622206 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.622333 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:27.623663 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:27.623668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:27.623671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:27.623688 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:27.623719 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:27.623730 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:27.623803 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:27.624041 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 01:16:27.624137 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937}
I20260812 01:16:27.625468 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:27.615076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:40211 (local address 127.20.151.3:46559)
0812 01:16:27.616110 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:27.616157 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:27.617909 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:27.618595 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:27.618603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:27.618691 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:27.619317 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:27.619337 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.620401 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.620405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:27.621664 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:27.621677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.622422 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.622431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:27.622661 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:27.623349 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:16:27.623352 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:27.623392 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:16:27.623481 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:27.623815 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:27.623827 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:27.623840 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 01:16:27.623997 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 01:16:27.624423 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":377,"threads_started":1}
I20260812 01:16:27.626827 20199 heartbeater.cc:347] Connected to a master server at 127.20.151.62:40211
I20260812 01:16:27.627148 20199 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:27.628003 20199 heartbeater.cc:511] Master 127.20.151.62:40211 requested a full tablet report, sending...
I20260812 01:16:27.629680 19407 ts_manager.cc:194] Registered new tserver with Master: 248590f8437b4debbdfb91f4f24235d9 (127.20.151.3:33035)
I20260812 01:16:27.630539 19407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.3:46559
I20260812 01:16:27.642405 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:16:27.649979 19713 heartbeater.cc:503] Master 127.20.151.62:40211 was elected leader, sending a full tablet report...
I20260812 01:16:27.653270 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:27.646775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.20.151.62:40211)
0812 01:16:27.646994 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:27.647000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:27.647319 (+ 319us) server_negotiation.cc:407] Connection header received
0812 01:16:27.647515 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:27.647521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:27.647624 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:27.647712 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:27.648228 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.649057 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.650703 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.651405 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.651532 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:27.652639 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:27.652644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:27.652647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:27.652664 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:27.652695 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:27.652707 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:27.652776 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:27.652860 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 01:16:27.652937 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:16:27.657255 19407 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:16:27.664237 19407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:16:27.703328 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:27.677228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.20.151.3:33035)
0812 01:16:27.693615 (+ 16387us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:27.693626 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:27.693666 (+ 40us) server_negotiation.cc:407] Connection header received
0812 01:16:27.693783 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:27.693791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:27.694108 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:27.694325 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:27.695180 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.697212 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.700597 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.702056 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.702205 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:27.702505 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:27.702761 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 01:16:27.702939 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16180,"thread_start_us":17338,"threads_started":1}
I20260812 01:16:27.703401 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:27.677117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:33035 (local address 127.0.0.1:35700)
0812 01:16:27.691421 (+ 14304us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:27.691447 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:27.691597 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:27.694379 (+ 2782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:27.694388 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:27.694428 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:27.694851 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:27.694866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.697424 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.697429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:27.700346 (+ 2917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:27.700372 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.702267 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.702274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:27.702491 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:27.702546 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 01:16:27.702705 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 01:16:27.702901 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14132,"mutex_wait_us":1,"thread_start_us":10597,"threads_started":1}
I20260812 01:16:27.705610 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:27.679072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:46441 (local address 127.0.0.1:55676)
0812 01:16:27.698450 (+ 19378us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:27.698474 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:27.698613 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:27.699314 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:27.699320 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:27.699348 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:27.699809 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:27.699823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.701439 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.701444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:27.703144 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:27.703158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.704912 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.704918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:27.705087 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:27.705118 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 01:16:27.705193 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 01:16:27.705316 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19242,"thread_start_us":19062,"threads_started":1}
I20260812 01:16:27.705768 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:27.679803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55676 (local address 127.20.151.2:46441)
0812 01:16:27.697915 (+ 18112us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:27.697926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:27.698602 (+ 676us) server_negotiation.cc:407] Connection header received
0812 01:16:27.698778 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:27.698787 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:27.699066 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:27.700033 (+ 967us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:27.700269 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.701308 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.703412 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.704731 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.704847 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:27.705088 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:27.705267 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 01:16:27.705442 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17924,"thread_start_us":17780,"threads_started":1}
I20260812 01:16:27.707135 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:27.676765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34857 (local address 127.0.0.1:41068)
0812 01:16:27.688169 (+ 11404us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:27.688211 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:27.688420 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:27.700058 (+ 11638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:27.700062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:27.700100 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:27.700449 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:27.700458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.702297 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.702302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:27.704644 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:27.704658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.706414 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.706421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:27.706608 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:27.706640 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:16:27.706730 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 01:16:27.706858 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11114,"thread_start_us":14918,"threads_started":1}
I20260812 01:16:27.707342 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:27.677015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.20.151.1:34857)
0812 01:16:27.699159 (+ 22144us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:27.699178 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:27.699207 (+ 29us) server_negotiation.cc:407] Connection header received
0812 01:16:27.699338 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:27.699346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:27.699558 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:27.699780 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:27.700651 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.702134 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.704826 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:27.706219 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:27.706360 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:27.706671 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:27.706787 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 01:16:27.706948 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21980,"thread_start_us":21846,"threads_started":1}
I20260812 01:16:27.709054 20087 tablet_service.cc:1467] Processing CreateTablet for tablet 38efc5b74aa440c0aa363b83adbbf493 (DEFAULT_TABLE table=test-table [id=92394dd47720472ba2dcd546f32b1ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:27.709437 19830 tablet_service.cc:1467] Processing CreateTablet for tablet 38efc5b74aa440c0aa363b83adbbf493 (DEFAULT_TABLE table=test-table [id=92394dd47720472ba2dcd546f32b1ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:27.710727 19830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38efc5b74aa440c0aa363b83adbbf493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:27.710826 20087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38efc5b74aa440c0aa363b83adbbf493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:27.710913 19604 tablet_service.cc:1467] Processing CreateTablet for tablet 38efc5b74aa440c0aa363b83adbbf493 (DEFAULT_TABLE table=test-table [id=92394dd47720472ba2dcd546f32b1ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:27.712328 19604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38efc5b74aa440c0aa363b83adbbf493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:27.723382 20231 tablet_bootstrap.cc:492] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: Bootstrap starting.
I20260812 01:16:27.726435 20231 tablet_bootstrap.cc:654] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:27.727366 20233 tablet_bootstrap.cc:492] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: Bootstrap starting.
I20260812 01:16:27.727837 20231 log.cc:824] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:27.728998 20236 tablet_bootstrap.cc:492] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1: Bootstrap starting.
I20260812 01:16:27.731163 20233 tablet_bootstrap.cc:654] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:27.732409 20236 tablet_bootstrap.cc:654] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:27.732498 20233 log.cc:824] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:27.733352 20231 tablet_bootstrap.cc:492] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: No bootstrap required, opened a new log
I20260812 01:16:27.733482 20236 log.cc:824] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:27.733634 20231 ts_tablet_manager.cc:1397] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20260812 01:16:27.735791 20233 tablet_bootstrap.cc:492] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: No bootstrap required, opened a new log
I20260812 01:16:27.736078 20233 ts_tablet_manager.cc:1397] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260812 01:16:27.741719 20236 tablet_bootstrap.cc:492] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1: No bootstrap required, opened a new log
I20260812 01:16:27.742134 20236 ts_tablet_manager.cc:1397] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260812 01:16:27.744696 20231 raft_consensus.cc:348] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d71fd633576459cad4c692fc7c5f3f1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34857 } } peers { permanent_uuid: "248590f8437b4debbdfb91f4f24235d9" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33035 } } peers { permanent_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46441 } }
I20260812 01:16:27.745201 20231 raft_consensus.cc:374] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:27.745371 20231 raft_consensus.cc:729] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e696d4eb4e6742a3ba4034f0771e08c9, State: Initialized, Role: FOLLOWER
I20260812 01:16:27.746054 20231 consensus_queue.cc:260] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d71fd633576459cad4c692fc7c5f3f1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34857 } } peers { permanent_uuid: "248590f8437b4debbdfb91f4f24235d9" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33035 } } peers { permanent_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46441 } }
I20260812 01:16:27.746567 20233 raft_consensus.cc:348] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d71fd633576459cad4c692fc7c5f3f1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34857 } } peers { permanent_uuid: "248590f8437b4debbdfb91f4f24235d9" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33035 } } peers { permanent_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46441 } }
I20260812 01:16:27.747092 20233 raft_consensus.cc:374] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:27.747219 20233 raft_consensus.cc:729] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248590f8437b4debbdfb91f4f24235d9, State: Initialized, Role: FOLLOWER
I20260812 01:16:27.747292 19954 heartbeater.cc:503] Master 127.20.151.62:40211 was elected leader, sending a full tablet report...
I20260812 01:16:27.747856 20233 consensus_queue.cc:260] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d71fd633576459cad4c692fc7c5f3f1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34857 } } peers { permanent_uuid: "248590f8437b4debbdfb91f4f24235d9" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33035 } } peers { permanent_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46441 } }
I20260812 01:16:27.749297 20199 heartbeater.cc:503] Master 127.20.151.62:40211 was elected leader, sending a full tablet report...
I20260812 01:16:27.749993 20233 ts_tablet_manager.cc:1428] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260812 01:16:27.751304 20231 ts_tablet_manager.cc:1428] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: Time spent starting tablet: real 0.018s user 0.012s sys 0.003s
I20260812 01:16:27.751924 20236 raft_consensus.cc:348] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d71fd633576459cad4c692fc7c5f3f1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34857 } } peers { permanent_uuid: "248590f8437b4debbdfb91f4f24235d9" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33035 } } peers { permanent_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46441 } }
I20260812 01:16:27.752336 20236 raft_consensus.cc:374] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:27.752434 20236 raft_consensus.cc:729] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d71fd633576459cad4c692fc7c5f3f1, State: Initialized, Role: FOLLOWER
I20260812 01:16:27.753041 20236 consensus_queue.cc:260] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d71fd633576459cad4c692fc7c5f3f1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34857 } } peers { permanent_uuid: "248590f8437b4debbdfb91f4f24235d9" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33035 } } peers { permanent_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46441 } }
I20260812 01:16:27.754676 20236 ts_tablet_manager.cc:1428] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1: Time spent starting tablet: real 0.012s user 0.005s sys 0.005s
W20260812 01:16:27.863365 20203 tablet.cc:2369] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:16:27.878360 19714 tablet.cc:2369] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:16:27.894163 19959 tablet.cc:2369] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:16:28.141999 20244 raft_consensus.cc:482] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:16:28.142349 20244 raft_consensus.cc:504] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d71fd633576459cad4c692fc7c5f3f1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34857 } } peers { permanent_uuid: "248590f8437b4debbdfb91f4f24235d9" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33035 } } peers { permanent_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46441 } }
I20260812 01:16:28.144151 20244 leader_election.cc:288] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d71fd633576459cad4c692fc7c5f3f1 (127.20.151.1:34857), e696d4eb4e6742a3ba4034f0771e08c9 (127.20.151.2:46441)
I20260812 01:16:28.152855 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.144837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:59089 (local address 127.20.151.1:34857)
0812 01:16:28.145063 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:28.145070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:28.145991 (+ 921us) server_negotiation.cc:407] Connection header received
0812 01:16:28.146299 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:28.146305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:28.146400 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:28.146468 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:28.147329 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.148313 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.150969 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.152028 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.152178 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:28.152249 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:28.152492 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 01:16:28.152618 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 01:16:28.152912 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.144514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34857 (local address 127.20.151.3:59089)
0812 01:16:28.145866 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:28.145889 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:28.146119 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:28.146676 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:28.146680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:28.146707 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:28.147068 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:28.147083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.148457 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.148466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:28.150763 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:28.150778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.152172 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.152178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:28.152347 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:28.152371 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 01:16:28.152449 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 01:16:28.152581 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":3,"thread_start_us":1114,"threads_started":1}
I20260812 01:16:28.153993 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.145246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:41057 (local address 127.20.151.2:46441)
0812 01:16:28.145453 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:28.145460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:28.147825 (+ 2365us) server_negotiation.cc:407] Connection header received
0812 01:16:28.147969 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:28.147975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:28.148075 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:28.148166 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:28.148927 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.149721 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.151657 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.152868 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.153007 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:28.153081 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:28.153577 (+ 496us) server_negotiation.cc:299] Negotiation successful
0812 01:16:28.153707 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 01:16:28.154194 19644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38efc5b74aa440c0aa363b83adbbf493" candidate_uuid: "248590f8437b4debbdfb91f4f24235d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d71fd633576459cad4c692fc7c5f3f1" is_pre_election: true
I20260812 01:16:28.154657 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.144612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:46441 (local address 127.20.151.3:41057)
0812 01:16:28.147708 (+ 3096us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:28.147727 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:28.147842 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:28.148196 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:28.148201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:28.148226 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:28.148753 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:28.148764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.149847 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.149853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:28.151476 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:28.151491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.153214 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.153221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:28.153397 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:28.153431 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 01:16:28.154240 (+ 809us) client_negotiation.cc:240] Negotiation successful
0812 01:16:28.154378 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2950,"thread_start_us":2764,"threads_started":1}
I20260812 01:16:28.154762 19644 raft_consensus.cc:2454] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248590f8437b4debbdfb91f4f24235d9 in term 0.
I20260812 01:16:28.155534 19997 leader_election.cc:302] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248590f8437b4debbdfb91f4f24235d9, 3d71fd633576459cad4c692fc7c5f3f1; no voters:
I20260812 01:16:28.155609 19873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38efc5b74aa440c0aa363b83adbbf493" candidate_uuid: "248590f8437b4debbdfb91f4f24235d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" is_pre_election: true
I20260812 01:16:28.155927 20244 raft_consensus.cc:2781] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:16:28.156103 20244 raft_consensus.cc:482] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:16:28.156147 19873 raft_consensus.cc:2454] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248590f8437b4debbdfb91f4f24235d9 in term 0.
I20260812 01:16:28.156217 20244 raft_consensus.cc:3037] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:28.158514 20244 raft_consensus.cc:504] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d71fd633576459cad4c692fc7c5f3f1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34857 } } peers { permanent_uuid: "248590f8437b4debbdfb91f4f24235d9" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33035 } } peers { permanent_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46441 } }
I20260812 01:16:28.159071 20244 leader_election.cc:288] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [CANDIDATE]: Term 1 election: Requested vote from peers 3d71fd633576459cad4c692fc7c5f3f1 (127.20.151.1:34857), e696d4eb4e6742a3ba4034f0771e08c9 (127.20.151.2:46441)
I20260812 01:16:28.159590 19873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38efc5b74aa440c0aa363b83adbbf493" candidate_uuid: "248590f8437b4debbdfb91f4f24235d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e696d4eb4e6742a3ba4034f0771e08c9"
I20260812 01:16:28.159619 19644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38efc5b74aa440c0aa363b83adbbf493" candidate_uuid: "248590f8437b4debbdfb91f4f24235d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d71fd633576459cad4c692fc7c5f3f1"
I20260812 01:16:28.159765 19873 raft_consensus.cc:3037] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:28.159816 19644 raft_consensus.cc:3037] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:28.162091 19644 raft_consensus.cc:2454] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248590f8437b4debbdfb91f4f24235d9 in term 1.
I20260812 01:16:28.162173 19873 raft_consensus.cc:2454] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248590f8437b4debbdfb91f4f24235d9 in term 1.
I20260812 01:16:28.162498 19997 leader_election.cc:302] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248590f8437b4debbdfb91f4f24235d9, 3d71fd633576459cad4c692fc7c5f3f1; no voters:
I20260812 01:16:28.162778 20244 raft_consensus.cc:2781] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:28.163239 20244 raft_consensus.cc:686] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 1 LEADER]: Becoming Leader. State: Replica: 248590f8437b4debbdfb91f4f24235d9, State: Running, Role: LEADER
I20260812 01:16:28.163743 20244 consensus_queue.cc:237] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d71fd633576459cad4c692fc7c5f3f1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34857 } } peers { permanent_uuid: "248590f8437b4debbdfb91f4f24235d9" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33035 } } peers { permanent_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46441 } }
I20260812 01:16:28.168695 19410 catalog_manager.cc:5495] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 248590f8437b4debbdfb91f4f24235d9 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "248590f8437b4debbdfb91f4f24235d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d71fd633576459cad4c692fc7c5f3f1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248590f8437b4debbdfb91f4f24235d9" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 33035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46441 } health_report { overall_health: UNKNOWN } } }
I20260812 01:16:28.269435 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.263578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.20.151.3:33035)
0812 01:16:28.264268 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:28.264273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:28.264291 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:16:28.264384 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:28.264388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:28.264476 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:28.264628 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:16:28.265045 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.265722 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.267144 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.267610 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.267702 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:28.268260 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:16:28.268840 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:16:28.268960 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:28.269080 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 01:16:28.269172 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":469,"threads_started":1}
I20260812 01:16:28.277907 19873 raft_consensus.cc:1261] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9 [term 1 FOLLOWER]: Refusing update from remote peer 248590f8437b4debbdfb91f4f24235d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:28.277907 19644 raft_consensus.cc:1261] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1 [term 1 FOLLOWER]: Refusing update from remote peer 248590f8437b4debbdfb91f4f24235d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:28.278964 20253 consensus_queue.cc:1035] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d71fd633576459cad4c692fc7c5f3f1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:16:28.279320 20244 consensus_queue.cc:1035] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e696d4eb4e6742a3ba4034f0771e08c9" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:16:28.290716 20263 mvcc.cc:205] Tried to move back new op lower bound from 7317493302375395328 to 7317493301922689024. Current Snapshot: MvccSnapshot[applied={T|T < 7317493302375395328}]
I20260812 01:16:28.303983 19407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:28.304234 19407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:28.307737 19407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bb89895d49c4bfaafce92971e0d303d: Sending DeleteTablet for 3 replicas of tablet 38efc5b74aa440c0aa363b83adbbf493
I20260812 01:16:28.308995 20087 tablet_service.cc:1514] Processing DeleteTablet for tablet 38efc5b74aa440c0aa363b83adbbf493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:28 UTC) from {username='slave'} at 127.0.0.1:35700
I20260812 01:16:28.308970 19604 tablet_service.cc:1514] Processing DeleteTablet for tablet 38efc5b74aa440c0aa363b83adbbf493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:28 UTC) from {username='slave'} at 127.0.0.1:41068
I20260812 01:16:28.309188 19830 tablet_service.cc:1514] Processing DeleteTablet for tablet 38efc5b74aa440c0aa363b83adbbf493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:28 UTC) from {username='slave'} at 127.0.0.1:55676
I20260812 01:16:28.311058 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 19508
I20260812 01:16:28.321686 20290 tablet_replica.cc:330] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: stopping tablet replica
I20260812 01:16:28.322417 20290 raft_consensus.cc:2229] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:16:28.323342 20290 raft_consensus.cc:2258] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:28.326174 20290 ts_tablet_manager.cc:1905] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:28.334483 20290 ts_tablet_manager.cc:1918] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:28.335247 20287 tablet_replica.cc:330] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1: stopping tablet replica
I20260812 01:16:28.335453 20290 log.cc:1196] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/38efc5b74aa440c0aa363b83adbbf493
I20260812 01:16:28.335529 20291 tablet_replica.cc:330] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: stopping tablet replica
I20260812 01:16:28.335978 20287 raft_consensus.cc:2229] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:28.335978 20291 raft_consensus.cc:2229] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:28.336159 20290 ts_tablet_manager.cc:1939] T 38efc5b74aa440c0aa363b83adbbf493 P 248590f8437b4debbdfb91f4f24235d9: Deleting consensus metadata
I20260812 01:16:28.336550 20287 raft_consensus.cc:2258] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:28.336552 20291 raft_consensus.cc:2258] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:28.338215 19384 catalog_manager.cc:4841] TS 248590f8437b4debbdfb91f4f24235d9 (127.20.151.3:33035): tablet 38efc5b74aa440c0aa363b83adbbf493 (table test-table [id=92394dd47720472ba2dcd546f32b1ed9]) successfully deleted
I20260812 01:16:28.338689 20291 ts_tablet_manager.cc:1905] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:28.338689 20287 ts_tablet_manager.cc:1905] T 38efc5b74aa440c0aa363b83adbbf493 P 3d71fd633576459cad4c692fc7c5f3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:28.341290 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.312511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.20.151.1:34857)
0812 01:16:28.312720 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:28.312727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:28.324324 (+ 11597us) server_negotiation.cc:407] Connection header received
0812 01:16:28.324515 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:28.324521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:28.324682 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:28.324789 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:28.327304 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.328417 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.332204 (+ 3787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.332856 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.332963 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:28.339311 (+ 6348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:28.339327 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:28.339346 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:28.340214 (+ 868us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:28.340285 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:28.340319 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:28.340417 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:28.340728 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 01:16:28.340871 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 01:16:28.342418 19707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:28.347496 20291 ts_tablet_manager.cc:1918] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:28.347712 20291 log.cc:1196] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/38efc5b74aa440c0aa363b83adbbf493
I20260812 01:16:28.348285 20291 ts_tablet_manager.cc:1939] T 38efc5b74aa440c0aa363b83adbbf493 P e696d4eb4e6742a3ba4034f0771e08c9: Deleting consensus metadata
I20260812 01:16:28.350502 19386 catalog_manager.cc:4841] TS e696d4eb4e6742a3ba4034f0771e08c9 (127.20.151.2:46441): tablet 38efc5b74aa440c0aa363b83adbbf493 (table test-table [id=92394dd47720472ba2dcd546f32b1ed9]) successfully deleted
I20260812 01:16:28.433825 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 19508
I20260812 01:16:28.462738 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 19717
W20260812 01:16:28.462728 19381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.151.1:34857 (error 108)
I20260812 01:16:28.468221 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.465462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34857 (local address 127.0.0.1:41082)
0812 01:16:28.467832 (+ 2370us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:28.467980 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2228,"thread_start_us":2051,"threads_started":1}
W20260812 01:16:28.468603 19381 catalog_manager.cc:4584] TS 3d71fd633576459cad4c692fc7c5f3f1 (127.20.151.1:34857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38efc5b74aa440c0aa363b83adbbf493: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34857: connect: Connection refused (error 111)
I20260812 01:16:28.471380 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.463939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.20.151.2:46441)
0812 01:16:28.464138 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:28.464144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:28.464164 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:16:28.464234 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:28.464240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:28.464372 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:28.464489 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:28.465020 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.466141 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.468078 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.468585 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.468706 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:28.469946 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:28.469956 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:28.469972 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:28.470585 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:28.470643 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:28.470666 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:28.470756 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:28.470911 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 01:16:28.471009 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 01:16:28.472124 19943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:28.522527 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.521942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34857 (local address 127.0.0.1:41086)
0812 01:16:28.522219 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:28.522342 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 01:16:28.523775 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.523343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34857 (local address 127.0.0.1:41102)
0812 01:16:28.523499 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:28.523592 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 01:16:28.562176 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 19717
I20260812 01:16:28.584607 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.583998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34857 (local address 127.0.0.1:41104)
0812 01:16:28.584277 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:28.584396 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260812 01:16:28.585868 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.585436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34857 (local address 127.0.0.1:41114)
0812 01:16:28.585600 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:28.585693 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 01:16:28.596100 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 19963
I20260812 01:16:28.603848 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.596856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35720 (local address 127.20.151.3:33035)
0812 01:16:28.597058 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:28.597064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:28.597126 (+ 62us) server_negotiation.cc:407] Connection header received
0812 01:16:28.597316 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:28.597322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:28.597426 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:28.597512 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:28.597929 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.598730 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.600159 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:28.600806 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:28.600953 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:28.601952 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:28.601965 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:28.601996 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:28.602880 (+ 884us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:28.602947 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:28.602994 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:28.603117 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:28.603241 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 01:16:28.603471 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 01:16:28.604678 20192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:28.666851 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.666324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34857 (local address 127.0.0.1:41126)
0812 01:16:28.666591 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:28.666693 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 01:16:28.667979 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:28.667544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:34857 (local address 127.0.0.1:41128)
0812 01:16:28.667741 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:28.667817 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.1:34857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 01:16:28.729648 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 19963
I20260812 01:16:28.756857 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 19352
I20260812 01:16:28.758001 19464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:28.869059 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 19352
2026-08-12T01:16:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3406 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:16:29 dist-test-slave-21b9 krb5kdc[20307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:16:29 dist-test-slave-21b9 krb5kdc[20307](info): set up 2 sockets
Aug 12 01:16:29 dist-test-slave-21b9 krb5kdc[20307](info): commencing operation
krb5kdc: starting...
W20260812 01:16:30.182428 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.004s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:16:30 dist-test-slave-21b9 krb5kdc[20307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:16:30.542171 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43967
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:43967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:30.895238 20329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:30.895635 20329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:30.895782 20329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:30.903117 20329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:30.903249 20329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:30.903353 20329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:30.903432 20329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:30.903496 20329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:30.913650 20329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:43967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:30.915024 20329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:30.916386 20329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:30.928125 20338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:30.928211 20340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:30.928256 20343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:30.929329 20329 server_base.cc:1034] running on GCE node
I20260812 01:16:30.930195 20329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:30.931679 20329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:30.932899 20329 hybrid_clock.cc:648] HybridClock initialized: now 1786497390932845 us; error 38 us; skew 500 ppm
I20260812 01:16:30.933238 20329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:30 dist-test-slave-21b9 krb5kdc[20307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497390, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:30.936439 20329 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:16:30.937933 20329 webserver.cc:460] Webserver started at http://127.20.151.62:32925/ using document root <none> and password file <none>
I20260812 01:16:30.938611 20329 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:30.938714 20329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:30.938974 20329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:30.942044 20329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "3f1b8e7e223b49cfad50dd8837df88a8"
format_stamp: "Formatted at 2026-08-12 01:16:30 on dist-test-slave-21b9"
server_key: "1f85b356a964168ef46182ff1ffd869d"
server_key_iv: "f965f97ef246492b9bd3ae7447c507b3"
server_key_version: "encryptionkey@0"
I20260812 01:16:30.942780 20329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "3f1b8e7e223b49cfad50dd8837df88a8"
format_stamp: "Formatted at 2026-08-12 01:16:30 on dist-test-slave-21b9"
server_key: "1f85b356a964168ef46182ff1ffd869d"
server_key_iv: "f965f97ef246492b9bd3ae7447c507b3"
server_key_version: "encryptionkey@0"
I20260812 01:16:30.948206 20329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 01:16:30.951874 20359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:30.953329 20329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 01:16:30.953504 20329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "3f1b8e7e223b49cfad50dd8837df88a8"
format_stamp: "Formatted at 2026-08-12 01:16:30 on dist-test-slave-21b9"
server_key: "1f85b356a964168ef46182ff1ffd869d"
server_key_iv: "f965f97ef246492b9bd3ae7447c507b3"
server_key_version: "encryptionkey@0"
I20260812 01:16:30.953691 20329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:30.967854 20329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:30.972273 20329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:30.972524 20329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:30.990283 20329 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:43967
I20260812 01:16:30.990307 20449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:43967 every 8 connection(s)
I20260812 01:16:30.991716 20329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:30.995564 20450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:30.996794 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 20329
I20260812 01:16:30.996986 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:30.997485 21084 external_mini_cluster.cc:1428] Setting key 35af997c834e3ca4de4ba8d535d7acb7
I20260812 01:16:31.010633 20450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8: Bootstrap starting.
Aug 12 01:16:31 dist-test-slave-21b9 krb5kdc[20307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:31.014006 20450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:31.015051 20450 log.cc:824] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:31.018123 20450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8: No bootstrap required, opened a new log
I20260812 01:16:31.022569 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:30.999395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60022 (local address 127.20.151.62:43967)
0812 01:16:31.002140 (+ 2745us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:31.002153 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:31.002200 (+ 47us) server_negotiation.cc:407] Connection header received
0812 01:16:31.003971 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:31.004004 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:31.004491 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:31.004880 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:31.006695 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:31.008221 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:31.009694 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:31.010399 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:31.010597 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:31.014971 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:31.014979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:31.014995 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:31.015035 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:31.018206 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:31.019120 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:31.019127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:31.019133 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:31.019344 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:31.019793 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:31.019796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:31.019798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:31.020361 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:31.020655 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:31.021083 (+ 428us) server_negotiation.cc:299] Negotiation successful
0812 01:16:31.021521 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2517,"thread_start_us":2278,"threads_started":1}
I20260812 01:16:31.027352 20450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b8e7e223b49cfad50dd8837df88a8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43967 } }
I20260812 01:16:31.027806 20450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:31.027922 20450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1b8e7e223b49cfad50dd8837df88a8, State: Initialized, Role: FOLLOWER
I20260812 01:16:31.028467 20450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b8e7e223b49cfad50dd8837df88a8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43967 } }
I20260812 01:16:31.028687 20450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:31.028784 20450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:31.028908 20450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:31.031046 20450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b8e7e223b49cfad50dd8837df88a8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43967 } }
I20260812 01:16:31.031551 20450 leader_election.cc:302] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f1b8e7e223b49cfad50dd8837df88a8; no voters:
I20260812 01:16:31.032110 20450 leader_election.cc:288] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:31.032241 20458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:31.032979 20458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [term 1 LEADER]: Becoming Leader. State: Replica: 3f1b8e7e223b49cfad50dd8837df88a8, State: Running, Role: LEADER
I20260812 01:16:31.033512 20458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b8e7e223b49cfad50dd8837df88a8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43967 } }
I20260812 01:16:31.034169 20450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:31.036384 20460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f1b8e7e223b49cfad50dd8837df88a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b8e7e223b49cfad50dd8837df88a8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43967 } } }
I20260812 01:16:31.036391 20461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f1b8e7e223b49cfad50dd8837df88a8. Latest consensus state: current_term: 1 leader_uuid: "3f1b8e7e223b49cfad50dd8837df88a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b8e7e223b49cfad50dd8837df88a8" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43967 } } }
I20260812 01:16:31.036818 20460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:31.036818 20461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:31.038314 20467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:31.045231 20467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:31.054992 20467 catalog_manager.cc:1347] Generated new cluster ID: 363581ff3c994d9897eb0b5a187b40da
I20260812 01:16:31.055119 20467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:31.068588 20467 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:31.070550 20467 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:31.084607 20467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8: Generated new TSK 0
I20260812 01:16:31.085534 20467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:16:31.182646 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43967
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:31.542193 20497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:31.542495 20497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:31.542603 20497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:31.550158 20497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:31.550284 20497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:31.550403 20497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:16:31.560802 20497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:31.562165 20497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:31.563536 20497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:31.577150 20504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:31.577236 20503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:31.577205 20510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:31.577956 20497 server_base.cc:1034] running on GCE node
I20260812 01:16:31.578552 20497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:31.579548 20497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:31.580752 20497 hybrid_clock.cc:648] HybridClock initialized: now 1786497391580694 us; error 52 us; skew 500 ppm
I20260812 01:16:31.581074 20497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:31 dist-test-slave-21b9 krb5kdc[20307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497391, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:31.584259 20497 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:16:31.585793 20497 webserver.cc:460] Webserver started at http://127.20.151.1:45463/ using document root <none> and password file <none>
I20260812 01:16:31.586500 20497 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:31.586624 20497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:31.586954 20497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:31.589944 20497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "8b4ca36d4a7346f19276449d546bfaa7"
format_stamp: "Formatted at 2026-08-12 01:16:31 on dist-test-slave-21b9"
server_key: "59df2651f1cc0be196e12941820cb025"
server_key_iv: "9b0e104b28924fac20cd57ce57471c67"
server_key_version: "encryptionkey@0"
I20260812 01:16:31.590713 20497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "8b4ca36d4a7346f19276449d546bfaa7"
format_stamp: "Formatted at 2026-08-12 01:16:31 on dist-test-slave-21b9"
server_key: "59df2651f1cc0be196e12941820cb025"
server_key_iv: "9b0e104b28924fac20cd57ce57471c67"
server_key_version: "encryptionkey@0"
I20260812 01:16:31.595980 20497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:16:31.599606 20524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:31.600915 20497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:16:31.601099 20497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "8b4ca36d4a7346f19276449d546bfaa7"
format_stamp: "Formatted at 2026-08-12 01:16:31 on dist-test-slave-21b9"
server_key: "59df2651f1cc0be196e12941820cb025"
server_key_iv: "9b0e104b28924fac20cd57ce57471c67"
server_key_version: "encryptionkey@0"
I20260812 01:16:31.601296 20497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:31.630288 20497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:31.634604 20497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:31.634920 20497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:31.636013 20497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:31.637429 20497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:31.637517 20497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:31.637650 20497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:31.637701 20497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:31.670403 20497 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:36915
I20260812 01:16:31.670439 20719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:36915 every 8 connection(s)
I20260812 01:16:31.671700 20497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:16:31.678593 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 20497
I20260812 01:16:31.678776 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:16:31.679242 21084 external_mini_cluster.cc:1428] Setting key 73f50c7bdbe621cbbccb036ba8269a0f
Aug 12 01:16:31 dist-test-slave-21b9 krb5kdc[20307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497391, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:31.692497 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:31.674271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:57569 (local address 127.20.151.62:43967)
0812 01:16:31.674747 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:31.674753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:31.675742 (+ 989us) server_negotiation.cc:407] Connection header received
0812 01:16:31.677598 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:31.677602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:31.677675 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:31.677767 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:31.679360 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:31.680309 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:31.682330 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:31.682879 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:31.683003 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:31.686467 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:31.686473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:31.686477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:31.686513 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:31.689415 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:31.690138 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:31.690143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:31.690146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:31.690288 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:31.690733 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:31.690737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:31.690740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:31.691080 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:31.691217 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:31.691949 (+ 732us) server_negotiation.cc:299] Negotiation successful
0812 01:16:31.692170 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":211,"threads_started":1}
I20260812 01:16:31.693415 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:31.674817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:43967 (local address 127.20.151.1:57569)
0812 01:16:31.675591 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:31.675629 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:31.677348 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:31.677955 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:31.677963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:31.678489 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:31.679149 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:31.679170 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:31.680483 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:31.680488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:31.682180 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:31.682193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:31.683102 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:31.683107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:31.683345 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:31.684008 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:16:31.684032 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:31.686241 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:31.689564 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:31.689572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:31.689590 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:31.690025 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:31.690394 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:31.690397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:31.690400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:31.690631 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:31.691189 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:31.691195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:31.691582 (+ 387us) client_negotiation.cc:769] Sending connection context
0812 01:16:31.691828 (+ 246us) client_negotiation.cc:240] Negotiation successful
0812 01:16:31.692243 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":274,"threads_started":1}
I20260812 01:16:31.694960 20720 heartbeater.cc:347] Connected to a master server at 127.20.151.62:43967
I20260812 01:16:31.695258 20720 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:31.696009 20720 heartbeater.cc:511] Master 127.20.151.62:43967 requested a full tablet report, sending...
I20260812 01:16:31.698077 20383 ts_manager.cc:194] Registered new tserver with Master: 8b4ca36d4a7346f19276449d546bfaa7 (127.20.151.1:36915)
I20260812 01:16:31.699968 20383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:57569
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:16:31.756825 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43967
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:32.115801 20727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:32.116096 20727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:32.116206 20727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:32.123546 20727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:32.123661 20727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:32.123782 20727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:16:32.133989 20727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:32.135461 20727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:32.136830 20727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:32.150112 20738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:32.150131 20739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:32.150200 20742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:32.151083 20727 server_base.cc:1034] running on GCE node
I20260812 01:16:32.151753 20727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:32.152736 20727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:32.153952 20727 hybrid_clock.cc:648] HybridClock initialized: now 1786497392153899 us; error 39 us; skew 500 ppm
I20260812 01:16:32.154294 20727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:32 dist-test-slave-21b9 krb5kdc[20307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497392, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:32.157696 20727 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:16:32.159293 20727 webserver.cc:460] Webserver started at http://127.20.151.2:36141/ using document root <none> and password file <none>
I20260812 01:16:32.160008 20727 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:32.160135 20727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:32.160451 20727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:32.163482 20727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "2d22c665f7be4921b9880a3f8c409bae"
format_stamp: "Formatted at 2026-08-12 01:16:32 on dist-test-slave-21b9"
server_key: "a998d5d79ddfc37146310d72bd4b655c"
server_key_iv: "c2af031586d22573eb94db20329e55bc"
server_key_version: "encryptionkey@0"
I20260812 01:16:32.164273 20727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "2d22c665f7be4921b9880a3f8c409bae"
format_stamp: "Formatted at 2026-08-12 01:16:32 on dist-test-slave-21b9"
server_key: "a998d5d79ddfc37146310d72bd4b655c"
server_key_iv: "c2af031586d22573eb94db20329e55bc"
server_key_version: "encryptionkey@0"
I20260812 01:16:32.169711 20727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:16:32.173573 20752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:32.175002 20727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:16:32.175168 20727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "2d22c665f7be4921b9880a3f8c409bae"
format_stamp: "Formatted at 2026-08-12 01:16:32 on dist-test-slave-21b9"
server_key: "a998d5d79ddfc37146310d72bd4b655c"
server_key_iv: "c2af031586d22573eb94db20329e55bc"
server_key_version: "encryptionkey@0"
I20260812 01:16:32.175444 20727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:32.201471 20727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:32.206386 20727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:32.206698 20727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:32.207789 20727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:32.209190 20727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:32.209265 20727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:32.209378 20727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:32.209426 20727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:32.241535 20727 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:43563
I20260812 01:16:32.241603 20934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:43563 every 8 connection(s)
I20260812 01:16:32.242854 20727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:16:32.253149 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 20727
I20260812 01:16:32.253355 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:16:32.253899 21084 external_mini_cluster.cc:1428] Setting key 83b2fffdb7f5e95b6c1b275897614f76
Aug 12 01:16:32 dist-test-slave-21b9 krb5kdc[20307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497392, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:32.263768 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:32.245431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:50529 (local address 127.20.151.62:43967)
0812 01:16:32.245946 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:32.245953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:32.246884 (+ 931us) server_negotiation.cc:407] Connection header received
0812 01:16:32.248769 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:32.248773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:32.248845 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:32.248944 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:32.250412 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.251137 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.252669 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.253123 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.253213 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:32.257166 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:32.257172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:32.257176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:32.257214 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:32.260477 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:32.261336 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:32.261341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:32.261344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:32.261502 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:32.261949 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:32.261953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:32.261955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:32.262282 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:32.262417 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:32.263129 (+ 712us) server_negotiation.cc:299] Negotiation successful
0812 01:16:32.263398 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":257,"threads_started":1}
I20260812 01:16:32.264751 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:32.245940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:43967 (local address 127.20.151.2:50529)
0812 01:16:32.246725 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:32.246767 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:32.248494 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:32.249135 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:32.249156 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:32.249710 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:32.250271 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:32.250289 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.251257 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.251260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:32.252512 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:32.252525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.253356 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.253365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:32.253705 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:32.254412 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:16:32.254442 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:32.256923 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:32.260664 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:32.260677 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:32.260693 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:32.261198 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:32.261615 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:32.261619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:32.261621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:32.261840 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:32.262392 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:32.262398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:32.262761 (+ 363us) client_negotiation.cc:769] Sending connection context
0812 01:16:32.263015 (+ 254us) client_negotiation.cc:240] Negotiation successful
0812 01:16:32.263517 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":291,"threads_started":1}
I20260812 01:16:32.266304 20935 heartbeater.cc:347] Connected to a master server at 127.20.151.62:43967
I20260812 01:16:32.266671 20935 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:32.267560 20935 heartbeater.cc:511] Master 127.20.151.62:43967 requested a full tablet report, sending...
I20260812 01:16:32.269218 20383 ts_manager.cc:194] Registered new tserver with Master: 2d22c665f7be4921b9880a3f8c409bae (127.20.151.2:43563)
I20260812 01:16:32.270252 20383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:50529
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:16:32.331157 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43967
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:32.680416 20946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:32.680733 20946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:32.680882 20946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:32.688618 20946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:32.688762 20946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:32.688915 20946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:16:32.699115 20946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:32.700459 20946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:32.701802 20946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 01:16:32.703799 20720 heartbeater.cc:503] Master 127.20.151.62:43967 was elected leader, sending a full tablet report...
W20260812 01:16:32.715801 20958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:32.715965 20956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:32.715989 20963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:32.716991 20946 server_base.cc:1034] running on GCE node
I20260812 01:16:32.717604 20946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:32.718597 20946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:32.719810 20946 hybrid_clock.cc:648] HybridClock initialized: now 1786497392719750 us; error 50 us; skew 500 ppm
I20260812 01:16:32.720150 20946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:32 dist-test-slave-21b9 krb5kdc[20307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497392, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:32.723388 20946 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:16:32.724929 20946 webserver.cc:460] Webserver started at http://127.20.151.3:35053/ using document root <none> and password file <none>
I20260812 01:16:32.725615 20946 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:32.725721 20946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:32.725999 20946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:32.729094 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "c66e6e93ce1b4c12ad64b24022f75a88"
format_stamp: "Formatted at 2026-08-12 01:16:32 on dist-test-slave-21b9"
server_key: "8dadbbaec1d9a9b9d08e57050aaff97b"
server_key_iv: "a32d429dccbcd9e914cb68d596ae78b8"
server_key_version: "encryptionkey@0"
I20260812 01:16:32.729907 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "c66e6e93ce1b4c12ad64b24022f75a88"
format_stamp: "Formatted at 2026-08-12 01:16:32 on dist-test-slave-21b9"
server_key: "8dadbbaec1d9a9b9d08e57050aaff97b"
server_key_iv: "a32d429dccbcd9e914cb68d596ae78b8"
server_key_version: "encryptionkey@0"
I20260812 01:16:32.735180 20946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 01:16:32.738698 20974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:32.740128 20946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:16:32.740281 20946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "c66e6e93ce1b4c12ad64b24022f75a88"
format_stamp: "Formatted at 2026-08-12 01:16:32 on dist-test-slave-21b9"
server_key: "8dadbbaec1d9a9b9d08e57050aaff97b"
server_key_iv: "a32d429dccbcd9e914cb68d596ae78b8"
server_key_version: "encryptionkey@0"
I20260812 01:16:32.740466 20946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:32.770011 20946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:32.774564 20946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:32.774855 20946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:32.775902 20946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:32.777480 20946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:32.777568 20946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:32.777688 20946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:32.777752 20946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:32.810708 20946 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:45995
I20260812 01:16:32.810772 21166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:45995 every 8 connection(s)
I20260812 01:16:32.812259 20946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:16:32.817487 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 20946
I20260812 01:16:32.817652 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:16:32.818109 21084 external_mini_cluster.cc:1428] Setting key a7879184ebf38393faa47d2f2085d351
Aug 12 01:16:32 dist-test-slave-21b9 krb5kdc[20307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497392, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:32.833905 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:32.814926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:46013 (local address 127.20.151.62:43967)
0812 01:16:32.815429 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:32.815435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:32.816403 (+ 968us) server_negotiation.cc:407] Connection header received
0812 01:16:32.818471 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:32.818477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:32.818585 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:32.818679 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:32.820650 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.821502 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.823052 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.823513 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.823611 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:32.827336 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:32.827342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:32.827345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:32.827386 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:32.830234 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:32.831028 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:32.831032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:32.831035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:32.831184 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:32.831812 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:32.831818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:32.831821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:32.832181 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:32.832360 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:32.833266 (+ 906us) server_negotiation.cc:299] Negotiation successful
0812 01:16:32.833501 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":268,"threads_started":1}
I20260812 01:16:32.834884 21173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:32.815439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:43967 (local address 127.20.151.3:46013)
0812 01:16:32.816220 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:32.816261 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:32.818170 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:32.818955 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:32.818967 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:32.819643 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:32.820446 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:32.820474 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.821650 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.821654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:32.822931 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:32.822944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.823849 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.823855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:32.824099 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:32.824733 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:16:32.824757 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:32.827059 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:32.830406 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:32.830414 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:32.830432 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:32.830906 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:32.831357 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:32.831362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:32.831366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:32.831681 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:32.832340 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:32.832349 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:32.832802 (+ 453us) client_negotiation.cc:769] Sending connection context
0812 01:16:32.833119 (+ 317us) client_negotiation.cc:240] Negotiation successful
0812 01:16:32.833632 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":294,"threads_started":1}
I20260812 01:16:32.836306 21167 heartbeater.cc:347] Connected to a master server at 127.20.151.62:43967
I20260812 01:16:32.836639 21167 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:32.837419 21167 heartbeater.cc:511] Master 127.20.151.62:43967 requested a full tablet report, sending...
I20260812 01:16:32.838934 20383 ts_manager.cc:194] Registered new tserver with Master: c66e6e93ce1b4c12ad64b24022f75a88 (127.20.151.3:45995)
I20260812 01:16:32.839871 20383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:46013
I20260812 01:16:32.846249 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:16:32.863655 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:32.850430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60034 (local address 127.20.151.62:43967)
0812 01:16:32.850688 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:32.850695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:32.851029 (+ 334us) server_negotiation.cc:407] Connection header received
0812 01:16:32.851186 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:32.851190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:32.851315 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:32.851401 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:32.852940 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.853722 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.855039 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.855478 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.855569 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:32.858359 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:32.858371 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:32.858376 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:32.858438 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:32.860956 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:32.861738 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:32.861743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:32.861745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:32.861868 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:32.862278 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:32.862281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:32.862284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:32.862621 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:32.862816 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:32.863122 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 01:16:32.863257 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:16:32.867908 20383 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:16:32.875304 20383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:16:32.912509 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:32.887915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:36915 (local address 127.0.0.1:46276)
0812 01:16:32.902548 (+ 14633us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:32.902594 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:32.902863 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:32.903871 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:32.903879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:32.903912 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:32.904385 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:32.904401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.906752 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.906758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:32.909372 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:32.909389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.911377 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.911384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:32.911656 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:32.911729 (+ 73us) client_negotiation.cc:769] Sending connection context
0812 01:16:32.911843 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 01:16:32.912039 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14359,"mutex_wait_us":1,"thread_start_us":14201,"threads_started":1}
I20260812 01:16:32.912818 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:32.888254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.20.151.1:36915)
0812 01:16:32.902061 (+ 13807us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:32.902071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:32.903133 (+ 1062us) server_negotiation.cc:407] Connection header received
0812 01:16:32.903252 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:32.903260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:32.903590 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:32.904625 (+ 1035us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:32.904894 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.906576 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.909588 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.911110 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.911979 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:32.912239 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:32.912317 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 01:16:32.912495 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13602,"thread_start_us":13402,"threads_started":1}
I20260812 01:16:32.914474 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:32.890302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:43563 (local address 127.0.0.1:56226)
0812 01:16:32.905912 (+ 15610us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:32.905928 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:32.906366 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:32.907666 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:32.907671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:32.907702 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:32.908238 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:32.908253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.909687 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.909692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:32.911776 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:32.911788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.913609 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.913626 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:32.913796 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:32.913831 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 01:16:32.913929 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 01:16:32.914062 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15506,"mutex_wait_us":86,"thread_start_us":14653,"threads_started":1}
I20260812 01:16:32.914480 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:32.890945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.20.151.2:43563)
0812 01:16:32.904721 (+ 13776us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:32.904730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:32.906140 (+ 1410us) server_negotiation.cc:407] Connection header received
0812 01:16:32.906982 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:32.906991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:32.907262 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:32.907483 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:32.908387 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.909509 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.911928 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.913396 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.913538 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:32.913737 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:32.913988 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 01:16:32.914165 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13603,"thread_start_us":13452,"threads_started":1}
I20260812 01:16:32.915071 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:32.888097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:45995 (local address 127.0.0.1:60646)
0812 01:16:32.905237 (+ 17140us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:32.905257 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:32.905446 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:32.907594 (+ 2148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:32.907598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:32.907625 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:32.908192 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:32.908215 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.910510 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.910515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:32.912680 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:32.912696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.914357 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.914364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:32.914532 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:32.914564 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:16:32.914648 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 01:16:32.914765 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17004,"thread_start_us":17564,"threads_started":1}
I20260812 01:16:32.915683 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:32.888714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.20.151.3:45995)
0812 01:16:32.906953 (+ 18239us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:32.906961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:32.906995 (+ 34us) server_negotiation.cc:407] Connection header received
0812 01:16:32.907092 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:32.907100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:32.907367 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:32.908438 (+ 1071us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:32.908701 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.910357 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.912856 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:32.914135 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:32.914846 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:32.915069 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:32.915152 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 01:16:32.915339 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18059,"thread_start_us":17908,"threads_started":1}
I20260812 01:16:32.917093 20608 tablet_service.cc:1467] Processing CreateTablet for tablet fb55de03982d4eb094bbf9a540ec9f3f (DEFAULT_TABLE table=test-table [id=e948da99a3694bcb8054af851775ee6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:32.918424 20608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb55de03982d4eb094bbf9a540ec9f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:32.919682 21053 tablet_service.cc:1467] Processing CreateTablet for tablet fb55de03982d4eb094bbf9a540ec9f3f (DEFAULT_TABLE table=test-table [id=e948da99a3694bcb8054af851775ee6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:32.920336 20835 tablet_service.cc:1467] Processing CreateTablet for tablet fb55de03982d4eb094bbf9a540ec9f3f (DEFAULT_TABLE table=test-table [id=e948da99a3694bcb8054af851775ee6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:32.921307 21053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb55de03982d4eb094bbf9a540ec9f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:32.921726 20835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb55de03982d4eb094bbf9a540ec9f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:32.930894 21207 tablet_bootstrap.cc:492] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: Bootstrap starting.
I20260812 01:16:32.934386 21207 tablet_bootstrap.cc:654] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:32.936331 21207 log.cc:824] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:32.936791 21208 tablet_bootstrap.cc:492] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: Bootstrap starting.
I20260812 01:16:32.937637 21211 tablet_bootstrap.cc:492] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: Bootstrap starting.
I20260812 01:16:32.940883 21208 tablet_bootstrap.cc:654] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:32.941066 21211 tablet_bootstrap.cc:654] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:32.941443 21207 tablet_bootstrap.cc:492] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: No bootstrap required, opened a new log
I20260812 01:16:32.941766 21207 ts_tablet_manager.cc:1397] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260812 01:16:32.941978 21208 log.cc:824] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:32.942323 21211 log.cc:824] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:32.945458 21211 tablet_bootstrap.cc:492] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: No bootstrap required, opened a new log
I20260812 01:16:32.945520 21208 tablet_bootstrap.cc:492] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: No bootstrap required, opened a new log
I20260812 01:16:32.945731 21211 ts_tablet_manager.cc:1397] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 01:16:32.945860 21208 ts_tablet_manager.cc:1397] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260812 01:16:32.953153 21211 raft_consensus.cc:348] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4ca36d4a7346f19276449d546bfaa7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36915 } } peers { permanent_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45995 } } peers { permanent_uuid: "2d22c665f7be4921b9880a3f8c409bae" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43563 } }
I20260812 01:16:32.953545 21211 raft_consensus.cc:374] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:32.953341 21207 raft_consensus.cc:348] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4ca36d4a7346f19276449d546bfaa7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36915 } } peers { permanent_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45995 } } peers { permanent_uuid: "2d22c665f7be4921b9880a3f8c409bae" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43563 } }
I20260812 01:16:32.953686 21211 raft_consensus.cc:729] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66e6e93ce1b4c12ad64b24022f75a88, State: Initialized, Role: FOLLOWER
I20260812 01:16:32.953840 21207 raft_consensus.cc:374] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:32.953938 21207 raft_consensus.cc:729] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4ca36d4a7346f19276449d546bfaa7, State: Initialized, Role: FOLLOWER
I20260812 01:16:32.954461 21211 consensus_queue.cc:260] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4ca36d4a7346f19276449d546bfaa7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36915 } } peers { permanent_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45995 } } peers { permanent_uuid: "2d22c665f7be4921b9880a3f8c409bae" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43563 } }
I20260812 01:16:32.954531 21207 consensus_queue.cc:260] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4ca36d4a7346f19276449d546bfaa7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36915 } } peers { permanent_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45995 } } peers { permanent_uuid: "2d22c665f7be4921b9880a3f8c409bae" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43563 } }
I20260812 01:16:32.956302 21208 raft_consensus.cc:348] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4ca36d4a7346f19276449d546bfaa7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36915 } } peers { permanent_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45995 } } peers { permanent_uuid: "2d22c665f7be4921b9880a3f8c409bae" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43563 } }
I20260812 01:16:32.956794 21208 raft_consensus.cc:374] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:32.956897 21208 raft_consensus.cc:729] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d22c665f7be4921b9880a3f8c409bae, State: Initialized, Role: FOLLOWER
I20260812 01:16:32.957556 21208 consensus_queue.cc:260] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4ca36d4a7346f19276449d546bfaa7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36915 } } peers { permanent_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45995 } } peers { permanent_uuid: "2d22c665f7be4921b9880a3f8c409bae" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43563 } }
I20260812 01:16:32.958950 20935 heartbeater.cc:503] Master 127.20.151.62:43967 was elected leader, sending a full tablet report...
I20260812 01:16:32.959700 21208 ts_tablet_manager.cc:1428] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260812 01:16:32.960657 21167 heartbeater.cc:503] Master 127.20.151.62:43967 was elected leader, sending a full tablet report...
I20260812 01:16:32.961238 21211 ts_tablet_manager.cc:1428] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260812 01:16:32.960188 21207 ts_tablet_manager.cc:1428] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: Time spent starting tablet: real 0.018s user 0.012s sys 0.007s
W20260812 01:16:32.995009 20936 tablet.cc:2369] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:16:33.064052 21171 tablet.cc:2369] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:16:33.081314 21222 raft_consensus.cc:482] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:16:33.081719 21222 raft_consensus.cc:504] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4ca36d4a7346f19276449d546bfaa7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36915 } } peers { permanent_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45995 } } peers { permanent_uuid: "2d22c665f7be4921b9880a3f8c409bae" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43563 } }
I20260812 01:16:33.083534 21222 leader_election.cc:288] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b4ca36d4a7346f19276449d546bfaa7 (127.20.151.1:36915), c66e6e93ce1b4c12ad64b24022f75a88 (127.20.151.3:45995)
I20260812 01:16:33.090462 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:33.084146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:43255 (local address 127.20.151.1:36915)
0812 01:16:33.084364 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:33.084371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:33.085057 (+ 686us) server_negotiation.cc:407] Connection header received
0812 01:16:33.085260 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:33.085264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:33.085326 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:33.085401 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:33.086007 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.086743 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.088658 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.089472 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.089626 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:33.089704 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:33.090074 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 01:16:33.090207 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 01:16:33.090461 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:33.084112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:38727 (local address 127.20.151.3:45995)
0812 01:16:33.084330 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:33.084336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:33.085149 (+ 813us) server_negotiation.cc:407] Connection header received
0812 01:16:33.085309 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:33.085313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:33.085375 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:33.085452 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:33.086096 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.086824 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.088664 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.089450 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.089622 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:33.089703 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:33.090073 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 01:16:33.090218 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 01:16:33.090660 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:33.084160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:45995 (local address 127.20.151.2:38727)
0812 01:16:33.085022 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:33.085041 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:33.085168 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:33.085482 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:33.085486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:33.085509 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:33.085922 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:33.085933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.086988 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.086992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:33.088495 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:33.088509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.089707 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.089714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:33.089905 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:33.089944 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 01:16:33.090023 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 01:16:33.090204 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":69,"thread_start_us":502,"threads_started":1}
I20260812 01:16:33.090700 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:33.084062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:36915 (local address 127.20.151.2:43255)
0812 01:16:33.084929 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:33.084947 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:33.085109 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:33.085435 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:33.085440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:33.085463 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:33.085847 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:33.085858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.086871 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.086876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:33.088498 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:33.088511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.089741 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.089748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:33.089884 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:33.089938 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 01:16:33.090026 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 01:16:33.090194 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":28,"thread_start_us":497,"threads_started":1}
I20260812 01:16:33.091984 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb55de03982d4eb094bbf9a540ec9f3f" candidate_uuid: "2d22c665f7be4921b9880a3f8c409bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" is_pre_election: true
I20260812 01:16:33.091984 20649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb55de03982d4eb094bbf9a540ec9f3f" candidate_uuid: "2d22c665f7be4921b9880a3f8c409bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4ca36d4a7346f19276449d546bfaa7" is_pre_election: true
I20260812 01:16:33.092604 21098 raft_consensus.cc:2454] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d22c665f7be4921b9880a3f8c409bae in term 0.
I20260812 01:16:33.092593 20649 raft_consensus.cc:2454] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d22c665f7be4921b9880a3f8c409bae in term 0.
I20260812 01:16:33.093338 20756 leader_election.cc:302] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d22c665f7be4921b9880a3f8c409bae, 8b4ca36d4a7346f19276449d546bfaa7; no voters:
I20260812 01:16:33.093742 21222 raft_consensus.cc:2781] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:16:33.093885 21222 raft_consensus.cc:482] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:16:33.093988 21222 raft_consensus.cc:3037] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:33.096374 21222 raft_consensus.cc:504] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4ca36d4a7346f19276449d546bfaa7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36915 } } peers { permanent_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45995 } } peers { permanent_uuid: "2d22c665f7be4921b9880a3f8c409bae" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43563 } }
I20260812 01:16:33.097031 21222 leader_election.cc:288] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [CANDIDATE]: Term 1 election: Requested vote from peers 8b4ca36d4a7346f19276449d546bfaa7 (127.20.151.1:36915), c66e6e93ce1b4c12ad64b24022f75a88 (127.20.151.3:45995)
I20260812 01:16:33.097553 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb55de03982d4eb094bbf9a540ec9f3f" candidate_uuid: "2d22c665f7be4921b9880a3f8c409bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c66e6e93ce1b4c12ad64b24022f75a88"
I20260812 01:16:33.097563 20649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb55de03982d4eb094bbf9a540ec9f3f" candidate_uuid: "2d22c665f7be4921b9880a3f8c409bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4ca36d4a7346f19276449d546bfaa7"
I20260812 01:16:33.097777 21098 raft_consensus.cc:3037] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:33.097785 20649 raft_consensus.cc:3037] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:33.100099 21098 raft_consensus.cc:2454] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d22c665f7be4921b9880a3f8c409bae in term 1.
I20260812 01:16:33.100117 20649 raft_consensus.cc:2454] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d22c665f7be4921b9880a3f8c409bae in term 1.
I20260812 01:16:33.100523 20756 leader_election.cc:302] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d22c665f7be4921b9880a3f8c409bae, 8b4ca36d4a7346f19276449d546bfaa7, c66e6e93ce1b4c12ad64b24022f75a88; no voters:
I20260812 01:16:33.100864 21222 raft_consensus.cc:2781] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:33.101311 21222 raft_consensus.cc:686] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 1 LEADER]: Becoming Leader. State: Replica: 2d22c665f7be4921b9880a3f8c409bae, State: Running, Role: LEADER
I20260812 01:16:33.101796 21222 consensus_queue.cc:237] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4ca36d4a7346f19276449d546bfaa7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36915 } } peers { permanent_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45995 } } peers { permanent_uuid: "2d22c665f7be4921b9880a3f8c409bae" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43563 } }
I20260812 01:16:33.106709 20382 catalog_manager.cc:5495] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d22c665f7be4921b9880a3f8c409bae (127.20.151.2). New cstate: current_term: 1 leader_uuid: "2d22c665f7be4921b9880a3f8c409bae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4ca36d4a7346f19276449d546bfaa7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d22c665f7be4921b9880a3f8c409bae" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20260812 01:16:33.156441 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:33.150969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.20.151.2:43563)
0812 01:16:33.151152 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:33.151157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:33.151315 (+ 158us) server_negotiation.cc:407] Connection header received
0812 01:16:33.151571 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:33.151597 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:33.151689 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:33.151841 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:16:33.152197 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.152909 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.154256 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.154661 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.154750 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:33.155341 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:16:33.155889 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:16:33.156037 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:33.156167 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 01:16:33.156214 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 01:16:33.165235 20649 raft_consensus.cc:1261] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7 [term 1 FOLLOWER]: Refusing update from remote peer 2d22c665f7be4921b9880a3f8c409bae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:33.165238 21098 raft_consensus.cc:1261] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88 [term 1 FOLLOWER]: Refusing update from remote peer 2d22c665f7be4921b9880a3f8c409bae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:33.166181 21225 consensus_queue.cc:1035] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4ca36d4a7346f19276449d546bfaa7" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 36915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:16:33.166553 21222 consensus_queue.cc:1035] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [LEADER]: Connected to new peer: Peer: permanent_uuid: "c66e6e93ce1b4c12ad64b24022f75a88" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 01:16:33.174396 20721 tablet.cc:2369] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:16:33.194327 20381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:33.194648 20381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:33.198820 20381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f1b8e7e223b49cfad50dd8837df88a8: Sending DeleteTablet for 3 replicas of tablet fb55de03982d4eb094bbf9a540ec9f3f
I20260812 01:16:33.200237 20608 tablet_service.cc:1514] Processing DeleteTablet for tablet fb55de03982d4eb094bbf9a540ec9f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:33 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:46276
I20260812 01:16:33.200325 20835 tablet_service.cc:1514] Processing DeleteTablet for tablet fb55de03982d4eb094bbf9a540ec9f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:33 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:56226
I20260812 01:16:33.201114 21053 tablet_service.cc:1514] Processing DeleteTablet for tablet fb55de03982d4eb094bbf9a540ec9f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:33 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:60646
I20260812 01:16:33.201478 21245 tablet_replica.cc:330] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: stopping tablet replica
I20260812 01:16:33.201707 21246 tablet_replica.cc:330] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: stopping tablet replica
I20260812 01:16:33.201946 21245 raft_consensus.cc:2229] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:33.202108 21246 raft_consensus.cc:2229] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:16:33.202567 21245 raft_consensus.cc:2258] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:33.202805 21246 raft_consensus.cc:2258] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:33.203341 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 20497
I20260812 01:16:33.207132 21245 ts_tablet_manager.cc:1905] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:33.212121 21246 ts_tablet_manager.cc:1905] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:33.214834 21245 ts_tablet_manager.cc:1918] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:33.215127 21245 log.cc:1196] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/fb55de03982d4eb094bbf9a540ec9f3f
I20260812 01:16:33.215822 21245 ts_tablet_manager.cc:1939] T fb55de03982d4eb094bbf9a540ec9f3f P 8b4ca36d4a7346f19276449d546bfaa7: Deleting consensus metadata
I20260812 01:16:33.218202 20362 catalog_manager.cc:4841] TS 8b4ca36d4a7346f19276449d546bfaa7 (127.20.151.1:36915): tablet fb55de03982d4eb094bbf9a540ec9f3f (table test-table [id=e948da99a3694bcb8054af851775ee6e]) successfully deleted
I20260812 01:16:33.219297 21247 tablet_replica.cc:330] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: stopping tablet replica
I20260812 01:16:33.219873 21247 raft_consensus.cc:2229] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:33.220515 21247 raft_consensus.cc:2258] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:33.223317 21247 ts_tablet_manager.cc:1905] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:33.223876 21246 ts_tablet_manager.cc:1918] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:33.224232 21246 log.cc:1196] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/fb55de03982d4eb094bbf9a540ec9f3f
I20260812 01:16:33.224864 21246 ts_tablet_manager.cc:1939] T fb55de03982d4eb094bbf9a540ec9f3f P 2d22c665f7be4921b9880a3f8c409bae: Deleting consensus metadata
Aug 12 01:16:33 dist-test-slave-21b9 krb5kdc[20307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:16:33.226848 20367 catalog_manager.cc:4841] TS 2d22c665f7be4921b9880a3f8c409bae (127.20.151.2:43563): tablet fb55de03982d4eb094bbf9a540ec9f3f (table test-table [id=e948da99a3694bcb8054af851775ee6e]) successfully deleted
I20260812 01:16:33.230080 21247 ts_tablet_manager.cc:1918] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:33.230266 21247 log.cc:1196] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/fb55de03982d4eb094bbf9a540ec9f3f
I20260812 01:16:33.230917 21247 ts_tablet_manager.cc:1939] T fb55de03982d4eb094bbf9a540ec9f3f P c66e6e93ce1b4c12ad64b24022f75a88: Deleting consensus metadata
I20260812 01:16:33.233281 20361 catalog_manager.cc:4841] TS c66e6e93ce1b4c12ad64b24022f75a88 (127.20.151.3:45995): tablet fb55de03982d4eb094bbf9a540ec9f3f (table test-table [id=e948da99a3694bcb8054af851775ee6e]) successfully deleted
I20260812 01:16:33.235787 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:33.205076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46278 (local address 127.20.151.1:36915)
0812 01:16:33.205252 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:33.205258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:33.214305 (+ 9047us) server_negotiation.cc:407] Connection header received
0812 01:16:33.214853 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:33.214860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:33.214955 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:33.215034 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:33.217038 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.218244 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.220273 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.220915 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.221025 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:33.228306 (+ 7281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:33.228328 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:33.228350 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:33.228397 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:33.232567 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:33.233590 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:33.233597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:33.233600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:33.233746 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:33.234160 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:33.234163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:33.234165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:33.234562 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:33.234852 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:33.235177 (+ 325us) server_negotiation.cc:299] Negotiation successful
0812 01:16:33.235416 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 01:16:33.236511 20713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:33.343098 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 20497
I20260812 01:16:33.374233 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 20727
Aug 12 01:16:33 dist-test-slave-21b9 krb5kdc[20307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:16:33.395406 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:33.375024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.20.151.2:43563)
0812 01:16:33.375202 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:33.375207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:33.375243 (+ 36us) server_negotiation.cc:407] Connection header received
0812 01:16:33.375529 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:33.375533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:33.375623 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:33.375720 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:33.377163 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.378019 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.379532 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.380211 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.380342 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:33.388171 (+ 7829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:33.388187 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:33.388208 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:33.388263 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:33.391962 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:33.392818 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:33.392828 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:33.392831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:33.393013 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:33.393458 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:33.393463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:33.393466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:33.393946 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:33.394314 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:33.394588 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 01:16:33.394982 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 01:16:33.396206 20929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:33.512017 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 20727
I20260812 01:16:33.538018 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 20946
Aug 12 01:16:33 dist-test-slave-21b9 krb5kdc[20307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:16:33.553484 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:33.538834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.20.151.3:45995)
0812 01:16:33.539000 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:33.539004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:33.539052 (+ 48us) server_negotiation.cc:407] Connection header received
0812 01:16:33.539244 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:33.539249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:33.539347 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:33.539460 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:33.540894 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.541656 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.542947 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:33.543591 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:33.543693 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:33.548232 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:33.548252 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:33.548270 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:33.548306 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:33.550887 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:33.551652 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:33.551660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:33.551663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:33.551815 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:33.552173 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:33.552176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:33.552178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:33.552525 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:33.552759 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:33.553055 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 01:16:33.553229 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 01:16:33.554108 21161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:33.645526 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 20946
I20260812 01:16:33.669633 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 20329
I20260812 01:16:33.670809 20442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:33.784754 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 20329
2026-08-12T01:16:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4910 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T01:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:33Z Disabled control of system clock
I20260812 01:16:33.891512 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43129
--webserver_interface=127.20.151.62
--webserver_port=0
--builtin_ntp_servers=127.20.151.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:43129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:16:34.251411 21259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:34.251742 21259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:34.251842 21259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:34.259429 21259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:34.259558 21259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:34.259599 21259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:34.259672 21259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:34.259718 21259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:34.269248 21259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:43129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:34.270591 21259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:34.272418 21259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:34.284238 21268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:34.284257 21270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:34.284298 21276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:34.284997 21259 server_base.cc:1034] running on GCE node
I20260812 01:16:34.285805 21259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:34.287261 21259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:34.288502 21259 hybrid_clock.cc:648] HybridClock initialized: now 1786497394288451 us; error 42 us; skew 500 ppm
I20260812 01:16:34.288854 21259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:34.291194 21259 webserver.cc:460] Webserver started at http://127.20.151.62:37697/ using document root <none> and password file <none>
I20260812 01:16:34.291937 21259 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:34.292032 21259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:34.292358 21259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:34.295344 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "350c6874c1084dd990877a290632a718"
format_stamp: "Formatted at 2026-08-12 01:16:34 on dist-test-slave-21b9"
server_key: "3d796f8f6910d6a561432892b3cbd26d"
server_key_iv: "53393526893474c40d8f058bcf0e3e5a"
server_key_version: "encryptionkey@0"
I20260812 01:16:34.296195 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "350c6874c1084dd990877a290632a718"
format_stamp: "Formatted at 2026-08-12 01:16:34 on dist-test-slave-21b9"
server_key: "3d796f8f6910d6a561432892b3cbd26d"
server_key_iv: "53393526893474c40d8f058bcf0e3e5a"
server_key_version: "encryptionkey@0"
I20260812 01:16:34.301582 21259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:16:34.305401 21288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:34.306876 21259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:16:34.307081 21259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "350c6874c1084dd990877a290632a718"
format_stamp: "Formatted at 2026-08-12 01:16:34 on dist-test-slave-21b9"
server_key: "3d796f8f6910d6a561432892b3cbd26d"
server_key_iv: "53393526893474c40d8f058bcf0e3e5a"
server_key_version: "encryptionkey@0"
I20260812 01:16:34.307348 21259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:34.327064 21259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:34.328105 21259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:34.328410 21259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:34.347200 21259 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:43129
I20260812 01:16:34.347220 21372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:43129 every 8 connection(s)
I20260812 01:16:34.348637 21259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:34.352542 21374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:34.358402 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 21259
I20260812 01:16:34.358582 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:34.359053 21084 external_mini_cluster.cc:1428] Setting key 175345a5433afc8f4b6902b899e1f847
I20260812 01:16:34.363878 21374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718: Bootstrap starting.
I20260812 01:16:34.367059 21374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:34.368124 21374 log.cc:824] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:34.371527 21374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718: No bootstrap required, opened a new log
I20260812 01:16:34.372968 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:34.361028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.20.151.62:43129)
0812 01:16:34.362105 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:34.362118 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:34.362160 (+ 42us) server_negotiation.cc:407] Connection header received
0812 01:16:34.363937 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:34.363961 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:34.364437 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:34.364798 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:34.365353 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:34.366674 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:34.368149 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:34.369135 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:34.369365 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:34.370613 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:34.370621 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:34.370636 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:34.371189 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:34.371296 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:34.371325 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:34.371414 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:34.371684 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 01:16:34.372025 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":411,"threads_started":1}
I20260812 01:16:34.379650 21374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c6874c1084dd990877a290632a718" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43129 } }
I20260812 01:16:34.380040 21374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:34.380146 21374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350c6874c1084dd990877a290632a718, State: Initialized, Role: FOLLOWER
I20260812 01:16:34.380717 21374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c6874c1084dd990877a290632a718" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43129 } }
I20260812 01:16:34.380978 21374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:34.381117 21374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:34.381278 21374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:34.383620 21374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c6874c1084dd990877a290632a718" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43129 } }
I20260812 01:16:34.384130 21374 leader_election.cc:302] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 350c6874c1084dd990877a290632a718; no voters:
I20260812 01:16:34.384752 21374 leader_election.cc:288] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:34.384886 21380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:34.385628 21380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [term 1 LEADER]: Becoming Leader. State: Replica: 350c6874c1084dd990877a290632a718, State: Running, Role: LEADER
I20260812 01:16:34.386153 21380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c6874c1084dd990877a290632a718" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43129 } }
I20260812 01:16:34.386864 21374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:34.389101 21381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "350c6874c1084dd990877a290632a718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c6874c1084dd990877a290632a718" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43129 } } }
I20260812 01:16:34.389541 21381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:34.390019 21382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 350c6874c1084dd990877a290632a718. Latest consensus state: current_term: 1 leader_uuid: "350c6874c1084dd990877a290632a718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350c6874c1084dd990877a290632a718" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43129 } } }
I20260812 01:16:34.390323 21382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:34.392074 21392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:34.397270 21392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:34.407147 21392 catalog_manager.cc:1347] Generated new cluster ID: 9156e903da8a49258d2fa458a33152bb
I20260812 01:16:34.407313 21392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:34.435427 21392 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:34.436880 21392 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:34.443383 21392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718: Generated new TSK 0
I20260812 01:16:34.444445 21392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:16:34.458209 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43129
--builtin_ntp_servers=127.20.151.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:16:34.822399 21411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:34.822719 21411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:34.822810 21411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:34.830369 21411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:34.830508 21411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:34.830623 21411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:16:34.840322 21411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:34.841652 21411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:34.843467 21411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:34.856950 21418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:34.856909 21419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:34.856909 21422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:34.857793 21411 server_base.cc:1034] running on GCE node
I20260812 01:16:34.858355 21411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:34.859365 21411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:34.860599 21411 hybrid_clock.cc:648] HybridClock initialized: now 1786497394860543 us; error 45 us; skew 500 ppm
I20260812 01:16:34.860935 21411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:34.863322 21411 webserver.cc:460] Webserver started at http://127.20.151.1:39235/ using document root <none> and password file <none>
I20260812 01:16:34.864022 21411 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:34.864120 21411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:34.864439 21411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:34.867388 21411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "19da058855bf424aba2c7c5ca320d95e"
format_stamp: "Formatted at 2026-08-12 01:16:34 on dist-test-slave-21b9"
server_key: "1f65135e7ee0eea2878b12f1a179bce0"
server_key_iv: "4c52654c4fe140e204427f33955662ba"
server_key_version: "encryptionkey@0"
I20260812 01:16:34.868198 21411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "19da058855bf424aba2c7c5ca320d95e"
format_stamp: "Formatted at 2026-08-12 01:16:34 on dist-test-slave-21b9"
server_key: "1f65135e7ee0eea2878b12f1a179bce0"
server_key_iv: "4c52654c4fe140e204427f33955662ba"
server_key_version: "encryptionkey@0"
I20260812 01:16:34.873500 21411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:16:34.877166 21434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:34.878474 21411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:16:34.878639 21411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "19da058855bf424aba2c7c5ca320d95e"
format_stamp: "Formatted at 2026-08-12 01:16:34 on dist-test-slave-21b9"
server_key: "1f65135e7ee0eea2878b12f1a179bce0"
server_key_iv: "4c52654c4fe140e204427f33955662ba"
server_key_version: "encryptionkey@0"
I20260812 01:16:34.878852 21411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:34.895731 21411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:34.896591 21411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:34.896894 21411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:34.897912 21411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:34.899482 21411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:34.899587 21411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:34.899696 21411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:34.899770 21411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:34.932138 21411 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:41635
I20260812 01:16:34.932201 21610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:41635 every 8 connection(s)
I20260812 01:16:34.933490 21411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:16:34.935235 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 21411
I20260812 01:16:34.935423 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:16:34.936025 21084 external_mini_cluster.cc:1428] Setting key 354f397454cac488ada138db8b5396ca
I20260812 01:16:34.938805 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43129
--builtin_ntp_servers=127.20.151.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:16:34.947094 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:34.936215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:45119 (local address 127.20.151.62:43129)
0812 01:16:34.936712 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:34.936717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:34.938169 (+ 1452us) server_negotiation.cc:407] Connection header received
0812 01:16:34.940162 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:34.940168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:34.940250 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:34.940361 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:34.941416 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:34.942386 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:34.944206 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:34.944757 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:34.944882 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:34.946281 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:34.946286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:34.946290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:34.946314 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:34.946361 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:34.946376 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:34.946471 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:34.946638 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 01:16:34.946739 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":257,"threads_started":1}
I20260812 01:16:34.948051 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:34.936930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:43129 (local address 127.20.151.1:45119)
0812 01:16:34.938009 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:34.938053 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:34.939861 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:34.940567 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:34.940576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:34.940661 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:34.941247 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:34.941265 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:34.942548 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:34.942552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:34.944062 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:34.944075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:34.944980 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:34.944985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:34.945204 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:34.945907 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:16:34.945911 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:34.945976 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:16:34.946083 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:34.946464 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:34.946472 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:34.946477 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 01:16:34.946614 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 01:16:34.946922 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":344,"threads_started":1}
I20260812 01:16:34.949636 21611 heartbeater.cc:347] Connected to a master server at 127.20.151.62:43129
I20260812 01:16:34.949976 21611 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:34.950919 21611 heartbeater.cc:511] Master 127.20.151.62:43129 requested a full tablet report, sending...
I20260812 01:16:34.953236 21317 ts_manager.cc:194] Registered new tserver with Master: 19da058855bf424aba2c7c5ca320d95e (127.20.151.1:41635)
I20260812 01:16:34.954684 21317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.1:45119
W20260812 01:16:35.291052 21619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:35.291401 21619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:35.291543 21619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:35.298949 21619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:35.299062 21619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:35.299170 21619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:16:35.308550 21619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:35.309850 21619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:35.311829 21619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:35.325106 21629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:35.325029 21630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:35.325026 21633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:35.325997 21619 server_base.cc:1034] running on GCE node
I20260812 01:16:35.326601 21619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:35.327594 21619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:35.328804 21619 hybrid_clock.cc:648] HybridClock initialized: now 1786497395328741 us; error 50 us; skew 500 ppm
I20260812 01:16:35.329157 21619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:35.331506 21619 webserver.cc:460] Webserver started at http://127.20.151.2:40115/ using document root <none> and password file <none>
I20260812 01:16:35.332208 21619 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:35.332302 21619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:35.332610 21619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:35.335541 21619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "14e0d07b59264a449a9dedce511010d2"
format_stamp: "Formatted at 2026-08-12 01:16:35 on dist-test-slave-21b9"
server_key: "bf257817e050839b29fd109b9306b920"
server_key_iv: "b009832ee2673163eb1251cbfab39cf7"
server_key_version: "encryptionkey@0"
I20260812 01:16:35.336330 21619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "14e0d07b59264a449a9dedce511010d2"
format_stamp: "Formatted at 2026-08-12 01:16:35 on dist-test-slave-21b9"
server_key: "bf257817e050839b29fd109b9306b920"
server_key_iv: "b009832ee2673163eb1251cbfab39cf7"
server_key_version: "encryptionkey@0"
I20260812 01:16:35.341435 21619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:16:35.345036 21644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:35.346453 21619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:16:35.346654 21619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "14e0d07b59264a449a9dedce511010d2"
format_stamp: "Formatted at 2026-08-12 01:16:35 on dist-test-slave-21b9"
server_key: "bf257817e050839b29fd109b9306b920"
server_key_iv: "b009832ee2673163eb1251cbfab39cf7"
server_key_version: "encryptionkey@0"
I20260812 01:16:35.346907 21619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:35.368249 21619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:35.369346 21619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:35.369638 21619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:35.370673 21619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:35.372273 21619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:35.372371 21619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:35.372458 21619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:35.372514 21619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:35.404788 21619 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:45603
I20260812 01:16:35.404837 21835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:45603 every 8 connection(s)
I20260812 01:16:35.406119 21619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:16:35.416524 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 21619
I20260812 01:16:35.416745 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:16:35.417236 21084 external_mini_cluster.cc:1428] Setting key 950f523dca7aa9b103d73ab1b92c930a
I20260812 01:16:35.418922 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:35.408745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:34231 (local address 127.20.151.62:43129)
0812 01:16:35.408936 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:35.408941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:35.410408 (+ 1467us) server_negotiation.cc:407] Connection header received
0812 01:16:35.412286 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:35.412291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:35.412373 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:35.412468 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:35.413586 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.414362 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.415988 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.416574 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.416709 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:35.418094 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:35.418100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:35.418121 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:35.418142 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:35.418177 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:35.418189 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:35.418275 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:35.418495 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 01:16:35.418593 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:16:35.420163 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:35.409300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:43129 (local address 127.20.151.2:34231)
0812 01:16:35.410249 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:35.410286 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:35.411961 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:35.412728 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:35.412737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:35.412833 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:35.413441 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:35.413460 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.414487 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.414490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:35.415862 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:35.415876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.416823 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.416832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:35.417062 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:35.417755 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:16:35.417758 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:35.417802 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:16:35.417886 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:35.418281 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:35.418292 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:35.418300 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 01:16:35.418460 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 01:16:35.418790 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":297,"threads_started":1}
I20260812 01:16:35.420534 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43129
--builtin_ntp_servers=127.20.151.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:16:35.421592 21836 heartbeater.cc:347] Connected to a master server at 127.20.151.62:43129
I20260812 01:16:35.421937 21836 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:35.422887 21836 heartbeater.cc:511] Master 127.20.151.62:43129 requested a full tablet report, sending...
I20260812 01:16:35.424577 21317 ts_manager.cc:194] Registered new tserver with Master: 14e0d07b59264a449a9dedce511010d2 (127.20.151.2:45603)
I20260812 01:16:35.425400 21317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.2:34231
W20260812 01:16:35.772213 21840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:35.772527 21840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:35.772617 21840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:35.780000 21840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:35.780118 21840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:35.780229 21840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:16:35.789836 21840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:43129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:35.791193 21840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:35.793130 21840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:35.806663 21851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:35.806715 21858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:35.806694 21850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:35.807483 21840 server_base.cc:1034] running on GCE node
I20260812 01:16:35.808111 21840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:35.809115 21840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:35.810369 21840 hybrid_clock.cc:648] HybridClock initialized: now 1786497395810310 us; error 50 us; skew 500 ppm
I20260812 01:16:35.810745 21840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:35.813251 21840 webserver.cc:460] Webserver started at http://127.20.151.3:43511/ using document root <none> and password file <none>
I20260812 01:16:35.814016 21840 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:35.814141 21840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:35.814461 21840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:35.817414 21840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "bb0510e0c38347d18330bfdde3e02b43"
format_stamp: "Formatted at 2026-08-12 01:16:35 on dist-test-slave-21b9"
server_key: "4c7479a54e0ab5b5490bee4b93ffdccd"
server_key_iv: "f16bde0f056517e49fbd9117c6abad2d"
server_key_version: "encryptionkey@0"
I20260812 01:16:35.818212 21840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "bb0510e0c38347d18330bfdde3e02b43"
format_stamp: "Formatted at 2026-08-12 01:16:35 on dist-test-slave-21b9"
server_key: "4c7479a54e0ab5b5490bee4b93ffdccd"
server_key_iv: "f16bde0f056517e49fbd9117c6abad2d"
server_key_version: "encryptionkey@0"
I20260812 01:16:35.823570 21840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:16:35.827080 21868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:35.828454 21840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:16:35.828619 21840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "bb0510e0c38347d18330bfdde3e02b43"
format_stamp: "Formatted at 2026-08-12 01:16:35 on dist-test-slave-21b9"
server_key: "4c7479a54e0ab5b5490bee4b93ffdccd"
server_key_iv: "f16bde0f056517e49fbd9117c6abad2d"
server_key_version: "encryptionkey@0"
I20260812 01:16:35.828832 21840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:35.852387 21840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:35.853297 21840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:35.853595 21840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:35.854626 21840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:35.856236 21840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:35.856335 21840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:35.856451 21840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:35.856523 21840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:35.888764 21840 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:42235
I20260812 01:16:35.888825 22065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:42235 every 8 connection(s)
I20260812 01:16:35.890126 21840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:16:35.897631 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 21840
I20260812 01:16:35.897846 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:16:35.898327 21084 external_mini_cluster.cc:1428] Setting key 665e538f64209f9f6321c461b9d5f6e7
I20260812 01:16:35.903458 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:35.892969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:33383 (local address 127.20.151.62:43129)
0812 01:16:35.893165 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:35.893170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:35.894618 (+ 1448us) server_negotiation.cc:407] Connection header received
0812 01:16:35.896539 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:35.896544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:35.896629 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:35.896730 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:35.898050 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.898886 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.900777 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.901375 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.901480 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:35.902760 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:35.902765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:35.902768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:35.902786 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:35.902820 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:35.902832 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:35.902900 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:35.903060 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 01:16:35.903158 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 01:16:35.904395 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:35.893544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:43129 (local address 127.20.151.3:33383)
0812 01:16:35.894472 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:35.894509 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:35.896224 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:35.896939 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:35.896947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:35.897039 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:35.897834 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:35.897863 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.899092 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.899097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:35.900637 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:35.900649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.901579 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.901586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:35.901791 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:35.902445 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:16:35.902448 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:35.902488 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:16:35.902574 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:35.902904 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:35.902910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:35.902915 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 01:16:35.903045 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 01:16:35.903403 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":283,"threads_started":1}
I20260812 01:16:35.905699 22066 heartbeater.cc:347] Connected to a master server at 127.20.151.62:43129
I20260812 01:16:35.905974 22066 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:35.906701 22066 heartbeater.cc:511] Master 127.20.151.62:43129 requested a full tablet report, sending...
I20260812 01:16:35.908072 21317 ts_manager.cc:194] Registered new tserver with Master: bb0510e0c38347d18330bfdde3e02b43 (127.20.151.3:42235)
I20260812 01:16:35.908891 21317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.3:33383
I20260812 01:16:35.913794 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:16:35.924104 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:35.917991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.20.151.62:43129)
0812 01:16:35.918172 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:35.918177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:35.918576 (+ 399us) server_negotiation.cc:407] Connection header received
0812 01:16:35.918695 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:35.918699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:35.918763 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:35.918843 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:35.919352 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.920128 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.921560 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.922030 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.922148 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:35.923368 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:35.923373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:35.923377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:35.923396 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:35.923430 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:35.923442 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:35.923606 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:35.923815 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 01:16:35.923865 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:16:35.928490 21317 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:16:35.935565 21317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:16:35.958053 21611 heartbeater.cc:503] Master 127.20.151.62:43129 was elected leader, sending a full tablet report...
I20260812 01:16:35.971004 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:35.949181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.20.151.1:41635)
0812 01:16:35.960889 (+ 11708us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:35.960899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:35.961974 (+ 1075us) server_negotiation.cc:407] Connection header received
0812 01:16:35.962080 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:35.962087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:35.962467 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:35.962810 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:35.963519 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.965102 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.968582 (+ 3480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.969800 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.969933 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:35.970227 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:35.970449 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 01:16:35.970624 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11522,"thread_start_us":11372,"threads_started":1}
I20260812 01:16:35.971161 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:35.948293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:45603 (local address 127.0.0.1:47856)
0812 01:16:35.961387 (+ 13094us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:35.961428 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:35.961663 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:35.962598 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:35.962607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:35.962645 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:35.963238 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:35.963255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.965345 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.965351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:35.968220 (+ 2869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:35.968236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.970340 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.970347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:35.970555 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:35.970585 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 01:16:35.970660 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 01:16:35.970790 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12806,"mutex_wait_us":188,"thread_start_us":13118,"threads_started":1}
I20260812 01:16:35.971208 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:35.948671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:41635 (local address 127.0.0.1:38560)
0812 01:16:35.961678 (+ 13007us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:35.961697 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:35.961830 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:35.962812 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:35.962816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:35.962848 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:35.963309 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:35.963324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.965246 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.965251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:35.968391 (+ 3140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:35.968403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.969991 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.969997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:35.970190 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:35.970250 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 01:16:35.970364 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 01:16:35.970576 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12879,"mutex_wait_us":326,"thread_start_us":12268,"threads_started":1}
I20260812 01:16:35.972010 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:35.948517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.20.151.2:45603)
0812 01:16:35.961024 (+ 12507us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:35.961034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:35.961887 (+ 853us) server_negotiation.cc:407] Connection header received
0812 01:16:35.962025 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:35.962035 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:35.962313 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:35.963144 (+ 831us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:35.963522 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.965171 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.968455 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.969850 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.969982 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:35.970200 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:35.971510 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0812 01:16:35.971697 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12294,"thread_start_us":12515,"threads_started":1}
I20260812 01:16:35.975670 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:35.963505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:42235 (local address 127.0.0.1:56830)
0812 01:16:35.967838 (+ 4333us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:35.967854 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:35.967971 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:35.968593 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:35.968597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:35.968624 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:35.969025 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:35.969037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.971142 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.971151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:35.973548 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:35.973562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.974854 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.974861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:35.975040 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:35.975073 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 01:16:35.975158 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 01:16:35.975326 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4204,"mutex_wait_us":424,"thread_start_us":4037,"threads_started":1}
I20260812 01:16:35.975759 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:35.962244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.20.151.3:42235)
0812 01:16:35.967522 (+ 5278us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:35.967530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:35.968068 (+ 538us) server_negotiation.cc:407] Connection header received
0812 01:16:35.968166 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:35.968172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:35.968394 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:35.968592 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:35.969253 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.970966 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.973730 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:35.974653 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:35.974783 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:35.975053 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:35.975211 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 01:16:35.975407 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5091,"thread_start_us":4900,"threads_started":1}
I20260812 01:16:35.975775 21723 tablet_service.cc:1467] Processing CreateTablet for tablet f13239a289034379a42cfc50ec3d30b5 (DEFAULT_TABLE table=test-table [id=a1987519e20d4eba9de54cb5a5a95cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:35.975730 21502 tablet_service.cc:1467] Processing CreateTablet for tablet f13239a289034379a42cfc50ec3d30b5 (DEFAULT_TABLE table=test-table [id=a1987519e20d4eba9de54cb5a5a95cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:35.977349 21502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13239a289034379a42cfc50ec3d30b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:35.977357 21723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13239a289034379a42cfc50ec3d30b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:35.979135 21947 tablet_service.cc:1467] Processing CreateTablet for tablet f13239a289034379a42cfc50ec3d30b5 (DEFAULT_TABLE table=test-table [id=a1987519e20d4eba9de54cb5a5a95cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:35.980337 21947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13239a289034379a42cfc50ec3d30b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:35.992156 22100 tablet_bootstrap.cc:492] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: Bootstrap starting.
I20260812 01:16:35.995718 22103 tablet_bootstrap.cc:492] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: Bootstrap starting.
I20260812 01:16:35.996188 22100 tablet_bootstrap.cc:654] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:35.997349 22100 log.cc:824] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:35.999752 22103 tablet_bootstrap.cc:654] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:36.000855 22100 tablet_bootstrap.cc:492] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: No bootstrap required, opened a new log
I20260812 01:16:36.001144 22100 ts_tablet_manager.cc:1397] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260812 01:16:36.001176 22103 log.cc:824] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:36.004662 22103 tablet_bootstrap.cc:492] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: No bootstrap required, opened a new log
I20260812 01:16:36.004971 22102 tablet_bootstrap.cc:492] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: Bootstrap starting.
I20260812 01:16:36.005065 22103 ts_tablet_manager.cc:1397] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260812 01:16:36.008865 22102 tablet_bootstrap.cc:654] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:36.010257 22102 log.cc:824] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:36.011691 22100 raft_consensus.cc:348] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0d07b59264a449a9dedce511010d2" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45603 } } peers { permanent_uuid: "19da058855bf424aba2c7c5ca320d95e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41635 } } peers { permanent_uuid: "bb0510e0c38347d18330bfdde3e02b43" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42235 } }
I20260812 01:16:36.012313 22100 raft_consensus.cc:374] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:36.012554 22100 raft_consensus.cc:729] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0510e0c38347d18330bfdde3e02b43, State: Initialized, Role: FOLLOWER
I20260812 01:16:36.013373 22100 consensus_queue.cc:260] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0d07b59264a449a9dedce511010d2" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45603 } } peers { permanent_uuid: "19da058855bf424aba2c7c5ca320d95e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41635 } } peers { permanent_uuid: "bb0510e0c38347d18330bfdde3e02b43" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42235 } }
I20260812 01:16:36.021986 22066 heartbeater.cc:503] Master 127.20.151.62:43129 was elected leader, sending a full tablet report...
I20260812 01:16:36.024946 22100 ts_tablet_manager.cc:1428] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: Time spent starting tablet: real 0.024s user 0.015s sys 0.007s
I20260812 01:16:36.027201 22103 raft_consensus.cc:348] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0d07b59264a449a9dedce511010d2" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45603 } } peers { permanent_uuid: "19da058855bf424aba2c7c5ca320d95e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41635 } } peers { permanent_uuid: "bb0510e0c38347d18330bfdde3e02b43" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42235 } }
I20260812 01:16:36.027781 22103 raft_consensus.cc:374] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:36.027900 22103 raft_consensus.cc:729] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e0d07b59264a449a9dedce511010d2, State: Initialized, Role: FOLLOWER
I20260812 01:16:36.028671 22103 consensus_queue.cc:260] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0d07b59264a449a9dedce511010d2" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45603 } } peers { permanent_uuid: "19da058855bf424aba2c7c5ca320d95e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41635 } } peers { permanent_uuid: "bb0510e0c38347d18330bfdde3e02b43" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42235 } }
I20260812 01:16:36.031872 21836 heartbeater.cc:503] Master 127.20.151.62:43129 was elected leader, sending a full tablet report...
I20260812 01:16:36.032858 22103 ts_tablet_manager.cc:1428] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: Time spent starting tablet: real 0.028s user 0.012s sys 0.003s
I20260812 01:16:36.035490 22102 tablet_bootstrap.cc:492] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: No bootstrap required, opened a new log
I20260812 01:16:36.035828 22102 ts_tablet_manager.cc:1397] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.019s
I20260812 01:16:36.043534 22102 raft_consensus.cc:348] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0d07b59264a449a9dedce511010d2" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45603 } } peers { permanent_uuid: "19da058855bf424aba2c7c5ca320d95e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41635 } } peers { permanent_uuid: "bb0510e0c38347d18330bfdde3e02b43" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42235 } }
I20260812 01:16:36.043991 22102 raft_consensus.cc:374] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:36.044091 22102 raft_consensus.cc:729] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19da058855bf424aba2c7c5ca320d95e, State: Initialized, Role: FOLLOWER
I20260812 01:16:36.044689 22102 consensus_queue.cc:260] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0d07b59264a449a9dedce511010d2" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45603 } } peers { permanent_uuid: "19da058855bf424aba2c7c5ca320d95e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41635 } } peers { permanent_uuid: "bb0510e0c38347d18330bfdde3e02b43" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42235 } }
I20260812 01:16:36.046504 22102 ts_tablet_manager.cc:1428] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
W20260812 01:16:36.141738 22067 tablet.cc:2369] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:16:36.158267 21838 tablet.cc:2369] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:16:36.185936 21614 tablet.cc:2369] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:16:36.248556 22111 raft_consensus.cc:482] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:16:36.248898 22111 raft_consensus.cc:504] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0d07b59264a449a9dedce511010d2" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45603 } } peers { permanent_uuid: "19da058855bf424aba2c7c5ca320d95e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41635 } } peers { permanent_uuid: "bb0510e0c38347d18330bfdde3e02b43" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42235 } }
I20260812 01:16:36.250731 22111 leader_election.cc:288] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14e0d07b59264a449a9dedce511010d2 (127.20.151.2:45603), bb0510e0c38347d18330bfdde3e02b43 (127.20.151.3:42235)
I20260812 01:16:36.257678 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:36.251295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:34401 (local address 127.20.151.2:45603)
0812 01:16:36.251555 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:36.251561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:36.251916 (+ 355us) server_negotiation.cc:407] Connection header received
0812 01:16:36.252260 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:36.252266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:36.252355 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:36.252433 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:36.252990 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.253900 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.256007 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.256745 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.256897 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:36.256968 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:36.257285 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 01:16:36.257411 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 01:16:36.257689 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:36.251805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:38597 (local address 127.20.151.3:42235)
0812 01:16:36.252000 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:36.252005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:36.252204 (+ 199us) server_negotiation.cc:407] Connection header received
0812 01:16:36.252396 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:36.252400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:36.252500 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:36.252578 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:36.253225 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.253935 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.255986 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.256764 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.256943 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:36.257012 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:36.257313 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 01:16:36.257437 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 01:16:36.257756 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:36.250976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:45603 (local address 127.20.151.1:34401)
0812 01:16:36.251791 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:36.251814 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:36.252083 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:36.252469 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:36.252475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:36.252501 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:36.252848 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:36.252859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.254030 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.254035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:36.255804 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:36.255816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.256940 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.256946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:36.257125 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:36.257149 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 01:16:36.257229 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 01:16:36.257354 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":119,"thread_start_us":841,"threads_started":1}
I20260812 01:16:36.257803 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:36.251097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:42235 (local address 127.20.151.1:38597)
0812 01:16:36.252116 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:36.252135 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:36.252267 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:36.252628 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:36.252636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:36.252689 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:36.253050 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:36.253060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.254080 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.254086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:36.255792 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:36.255805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.256966 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.256973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:36.257168 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:36.257191 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 01:16:36.257267 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 01:16:36.257384 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":98,"spinlock_wait_cycles":1408,"thread_start_us":301,"threads_started":1}
I20260812 01:16:36.258805 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13239a289034379a42cfc50ec3d30b5" candidate_uuid: "19da058855bf424aba2c7c5ca320d95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0510e0c38347d18330bfdde3e02b43" is_pre_election: true
I20260812 01:16:36.259016 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13239a289034379a42cfc50ec3d30b5" candidate_uuid: "19da058855bf424aba2c7c5ca320d95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e0d07b59264a449a9dedce511010d2" is_pre_election: true
I20260812 01:16:36.259326 21986 raft_consensus.cc:2454] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19da058855bf424aba2c7c5ca320d95e in term 0.
I20260812 01:16:36.259557 21755 raft_consensus.cc:2454] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19da058855bf424aba2c7c5ca320d95e in term 0.
I20260812 01:16:36.260005 21435 leader_election.cc:302] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19da058855bf424aba2c7c5ca320d95e, bb0510e0c38347d18330bfdde3e02b43; no voters:
I20260812 01:16:36.260389 22111 raft_consensus.cc:2781] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:16:36.260556 22111 raft_consensus.cc:482] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:16:36.260667 22111 raft_consensus.cc:3037] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:36.262769 22111 raft_consensus.cc:504] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0d07b59264a449a9dedce511010d2" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45603 } } peers { permanent_uuid: "19da058855bf424aba2c7c5ca320d95e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41635 } } peers { permanent_uuid: "bb0510e0c38347d18330bfdde3e02b43" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42235 } }
I20260812 01:16:36.263396 22111 leader_election.cc:288] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [CANDIDATE]: Term 1 election: Requested vote from peers 14e0d07b59264a449a9dedce511010d2 (127.20.151.2:45603), bb0510e0c38347d18330bfdde3e02b43 (127.20.151.3:42235)
I20260812 01:16:36.263777 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13239a289034379a42cfc50ec3d30b5" candidate_uuid: "19da058855bf424aba2c7c5ca320d95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e0d07b59264a449a9dedce511010d2"
I20260812 01:16:36.263859 21986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13239a289034379a42cfc50ec3d30b5" candidate_uuid: "19da058855bf424aba2c7c5ca320d95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0510e0c38347d18330bfdde3e02b43"
I20260812 01:16:36.263969 21755 raft_consensus.cc:3037] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:36.264056 21986 raft_consensus.cc:3037] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:36.266309 21986 raft_consensus.cc:2454] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19da058855bf424aba2c7c5ca320d95e in term 1.
I20260812 01:16:36.266461 21755 raft_consensus.cc:2454] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19da058855bf424aba2c7c5ca320d95e in term 1.
I20260812 01:16:36.266723 21435 leader_election.cc:302] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19da058855bf424aba2c7c5ca320d95e, bb0510e0c38347d18330bfdde3e02b43; no voters:
I20260812 01:16:36.267031 22111 raft_consensus.cc:2781] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:36.267519 22111 raft_consensus.cc:686] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 1 LEADER]: Becoming Leader. State: Replica: 19da058855bf424aba2c7c5ca320d95e, State: Running, Role: LEADER
I20260812 01:16:36.268007 22111 consensus_queue.cc:237] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0d07b59264a449a9dedce511010d2" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45603 } } peers { permanent_uuid: "19da058855bf424aba2c7c5ca320d95e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41635 } } peers { permanent_uuid: "bb0510e0c38347d18330bfdde3e02b43" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42235 } }
I20260812 01:16:36.272964 21317 catalog_manager.cc:5495] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 19da058855bf424aba2c7c5ca320d95e (127.20.151.1). New cstate: current_term: 1 leader_uuid: "19da058855bf424aba2c7c5ca320d95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0d07b59264a449a9dedce511010d2" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19da058855bf424aba2c7c5ca320d95e" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 41635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb0510e0c38347d18330bfdde3e02b43" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42235 } health_report { overall_health: UNKNOWN } } }
I20260812 01:16:36.351823 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:36.345650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.20.151.1:41635)
0812 01:16:36.345857 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:36.345862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:36.345880 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:16:36.345965 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:36.345969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:36.346060 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:36.346185 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:16:36.346583 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.347346 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.349395 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.349827 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.349941 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:36.350646 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:16:36.351167 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:16:36.351359 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:36.351527 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 01:16:36.351572 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:16:36.360351 21755 raft_consensus.cc:1261] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2 [term 1 FOLLOWER]: Refusing update from remote peer 19da058855bf424aba2c7c5ca320d95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:36.360436 21986 raft_consensus.cc:1261] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43 [term 1 FOLLOWER]: Refusing update from remote peer 19da058855bf424aba2c7c5ca320d95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:36.361367 22111 consensus_queue.cc:1035] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb0510e0c38347d18330bfdde3e02b43" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:16:36.361742 22117 consensus_queue.cc:1035] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e0d07b59264a449a9dedce511010d2" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:16:36.374147 22123 mvcc.cc:205] Tried to move back new op lower bound from 7317493335481552896 to 7317493335117340672. Current Snapshot: MvccSnapshot[applied={T|T < 7317493335481552896}]
I20260812 01:16:36.388787 21315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:36.389060 21315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:36.392795 21315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 350c6874c1084dd990877a290632a718: Sending DeleteTablet for 3 replicas of tablet f13239a289034379a42cfc50ec3d30b5
I20260812 01:16:36.394063 21723 tablet_service.cc:1514] Processing DeleteTablet for tablet f13239a289034379a42cfc50ec3d30b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:36 UTC) from {username='slave'} at 127.0.0.1:47856
I20260812 01:16:36.394156 21502 tablet_service.cc:1514] Processing DeleteTablet for tablet f13239a289034379a42cfc50ec3d30b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:36 UTC) from {username='slave'} at 127.0.0.1:38560
I20260812 01:16:36.394196 21947 tablet_service.cc:1514] Processing DeleteTablet for tablet f13239a289034379a42cfc50ec3d30b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:36 UTC) from {username='slave'} at 127.0.0.1:56830
I20260812 01:16:36.396389 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 21411
I20260812 01:16:36.404650 22144 tablet_replica.cc:330] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: stopping tablet replica
I20260812 01:16:36.404881 22138 tablet_replica.cc:330] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: stopping tablet replica
I20260812 01:16:36.405378 22138 raft_consensus.cc:2229] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:36.405469 22144 raft_consensus.cc:2229] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:36.405913 22138 raft_consensus.cc:2258] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:36.406054 22144 raft_consensus.cc:2258] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:36.408349 22138 ts_tablet_manager.cc:1905] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:36.408352 22144 ts_tablet_manager.cc:1905] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:36.408931 22140 tablet_replica.cc:330] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: stopping tablet replica
I20260812 01:16:36.409356 22140 raft_consensus.cc:2229] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:16:36.410017 22140 raft_consensus.cc:2258] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:36.412022 22140 ts_tablet_manager.cc:1905] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:36.415781 22144 ts_tablet_manager.cc:1918] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:36.415994 22144 log.cc:1196] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/f13239a289034379a42cfc50ec3d30b5
I20260812 01:16:36.416114 22138 ts_tablet_manager.cc:1918] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:36.416275 22138 log.cc:1196] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/f13239a289034379a42cfc50ec3d30b5
I20260812 01:16:36.416633 22144 ts_tablet_manager.cc:1939] T f13239a289034379a42cfc50ec3d30b5 P bb0510e0c38347d18330bfdde3e02b43: Deleting consensus metadata
I20260812 01:16:36.416837 22138 ts_tablet_manager.cc:1939] T f13239a289034379a42cfc50ec3d30b5 P 14e0d07b59264a449a9dedce511010d2: Deleting consensus metadata
I20260812 01:16:36.418510 21292 catalog_manager.cc:4841] TS 14e0d07b59264a449a9dedce511010d2 (127.20.151.2:45603): tablet f13239a289034379a42cfc50ec3d30b5 (table test-table [id=a1987519e20d4eba9de54cb5a5a95cba]) successfully deleted
I20260812 01:16:36.420064 21289 catalog_manager.cc:4841] TS bb0510e0c38347d18330bfdde3e02b43 (127.20.151.3:42235): tablet f13239a289034379a42cfc50ec3d30b5 (table test-table [id=a1987519e20d4eba9de54cb5a5a95cba]) successfully deleted
I20260812 01:16:36.422765 22140 ts_tablet_manager.cc:1918] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:36.423027 22140 log.cc:1196] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/f13239a289034379a42cfc50ec3d30b5
I20260812 01:16:36.423660 22140 ts_tablet_manager.cc:1939] T f13239a289034379a42cfc50ec3d30b5 P 19da058855bf424aba2c7c5ca320d95e: Deleting consensus metadata
I20260812 01:16:36.425578 21289 catalog_manager.cc:4841] TS 19da058855bf424aba2c7c5ca320d95e (127.20.151.1:41635): tablet f13239a289034379a42cfc50ec3d30b5 (table test-table [id=a1987519e20d4eba9de54cb5a5a95cba]) successfully deleted
I20260812 01:16:36.433244 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:36.397842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.20.151.1:41635)
0812 01:16:36.398031 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:36.398038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:36.425268 (+ 27230us) server_negotiation.cc:407] Connection header received
0812 01:16:36.425496 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:36.425501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:36.425637 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:36.425759 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:36.426241 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.427488 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.429477 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.430181 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.430313 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:36.431654 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:36.431666 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:36.431684 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:36.432436 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:36.432510 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:36.432533 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:36.432616 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:36.432781 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 01:16:36.432870 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:16:36.434077 21604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:36.531638 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 21411
I20260812 01:16:36.558696 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 21619
I20260812 01:16:36.565483 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:36.559519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.20.151.2:45603)
0812 01:16:36.559711 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:36.559715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:36.559735 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:16:36.559860 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:36.559864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:36.559946 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:36.560025 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:36.560345 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.561077 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.562367 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.562821 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.562924 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:36.564131 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:36.564140 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:36.564154 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:36.564772 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:36.564820 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:36.564841 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:36.564927 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:36.565048 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 01:16:36.565142 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 01:16:36.566181 21826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:36.656435 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 21619
I20260812 01:16:36.680276 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 21840
I20260812 01:16:36.687440 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:36.681219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45270 (local address 127.20.151.3:42235)
0812 01:16:36.681419 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:36.681424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:36.681445 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:16:36.681628 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:36.681633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:36.681733 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:36.681820 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:36.682243 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.682950 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.684412 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:36.684819 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:36.684932 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:36.685947 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:36.685956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:36.685974 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:36.686556 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:36.686606 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:36.686631 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:36.686726 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:36.686858 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 01:16:36.686970 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:16:36.688113 22060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:36.776343 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 21840
I20260812 01:16:36.804479 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 21259
I20260812 01:16:36.805603 21365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:36.917505 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 21259
2026-08-12T01:16:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3132 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:16:37 dist-test-slave-21b9 krb5kdc[22158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:16:37 dist-test-slave-21b9 krb5kdc[22158](info): set up 2 sockets
Aug 12 01:16:37 dist-test-slave-21b9 krb5kdc[22158](info): commencing operation
krb5kdc: starting...
W20260812 01:16:38.222287 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:16:38 dist-test-slave-21b9 krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:16:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:16:38.587395 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:45823
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:45823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:38.949182 22178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:38.949549 22178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:38.949700 22178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:38.957345 22178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:38.957487 22178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:38.957572 22178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:38.957656 22178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:38.957718 22178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:38.968077 22178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:45823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:45823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:38.969539 22178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:38.970963 22178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:38.983088 22203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:38.983139 22202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:38.983124 22210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:38.984088 22178 server_base.cc:1034] running on GCE node
I20260812 01:16:38.984934 22178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:38.986413 22178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:38.987628 22178 hybrid_clock.cc:648] HybridClock initialized: now 1786497398987567 us; error 50 us; skew 500 ppm
I20260812 01:16:38.987998 22178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:38 dist-test-slave-21b9 krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497398, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:38.994191 22178 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:16:38.995921 22178 webserver.cc:460] Webserver started at http://127.20.151.62:35387/ using document root <none> and password file <none>
I20260812 01:16:38.996687 22178 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:38.996817 22178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:38.997133 22178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:39.000356 22178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "788c0d57e12149768b50df0d78c8bea7"
format_stamp: "Formatted at 2026-08-12 01:16:38 on dist-test-slave-21b9"
server_key: "a947dc43bf278f0f04f0deb718889256"
server_key_iv: "b5d2baf4596c56c0602e162393838ff8"
server_key_version: "encryptionkey@0"
I20260812 01:16:39.001128 22178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "788c0d57e12149768b50df0d78c8bea7"
format_stamp: "Formatted at 2026-08-12 01:16:38 on dist-test-slave-21b9"
server_key: "a947dc43bf278f0f04f0deb718889256"
server_key_iv: "b5d2baf4596c56c0602e162393838ff8"
server_key_version: "encryptionkey@0"
I20260812 01:16:39.006596 22178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:16:39.010421 22222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:39.011960 22178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 01:16:39.012161 22178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "788c0d57e12149768b50df0d78c8bea7"
format_stamp: "Formatted at 2026-08-12 01:16:38 on dist-test-slave-21b9"
server_key: "a947dc43bf278f0f04f0deb718889256"
server_key_iv: "b5d2baf4596c56c0602e162393838ff8"
server_key_version: "encryptionkey@0"
I20260812 01:16:39.012360 22178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:39.030248 22178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:39.034821 22178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:39.035095 22178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:39.053576 22178 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:45823
I20260812 01:16:39.053634 22297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:45823 every 8 connection(s)
I20260812 01:16:39.055091 22178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:39.059152 22300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:39.063891 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 22178
I20260812 01:16:39.064082 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:39.064567 21084 external_mini_cluster.cc:1428] Setting key 836df669950da5252edaf49d32a2b87c
I20260812 01:16:39.071987 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7: Bootstrap starting.
I20260812 01:16:39.075956 22300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:39.077277 22300 log.cc:824] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:39.080650 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7: No bootstrap required, opened a new log
Aug 12 01:16:39 dist-test-slave-21b9 krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:39.091482 22300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c0d57e12149768b50df0d78c8bea7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45823 } }
I20260812 01:16:39.092000 22307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:39.066545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.20.151.62:45823)
0812 01:16:39.067475 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:39.067491 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:39.067555 (+ 64us) server_negotiation.cc:407] Connection header received
0812 01:16:39.070186 (+ 2631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:39.070220 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:39.070807 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:39.071316 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:39.072908 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:39.074956 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:39.076490 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:39.077605 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:39.077796 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:39.082383 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:39.082396 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:39.082418 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:39.082482 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:39.086811 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:39.087779 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:39.087792 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:39.087803 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:39.088044 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:39.088451 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:39.088456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:39.088459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:39.089312 (+ 853us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:39.089654 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:39.090024 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 01:16:39.090517 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":357,"threads_started":1}
I20260812 01:16:39.092000 22300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:39.092202 22300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788c0d57e12149768b50df0d78c8bea7, State: Initialized, Role: FOLLOWER
I20260812 01:16:39.092809 22300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c0d57e12149768b50df0d78c8bea7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45823 } }
I20260812 01:16:39.093041 22300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:39.093190 22300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:39.093357 22300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:39.095999 22300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c0d57e12149768b50df0d78c8bea7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45823 } }
I20260812 01:16:39.096491 22300 leader_election.cc:302] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788c0d57e12149768b50df0d78c8bea7; no voters:
I20260812 01:16:39.097081 22300 leader_election.cc:288] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:39.097232 22311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:39.097900 22311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [term 1 LEADER]: Becoming Leader. State: Replica: 788c0d57e12149768b50df0d78c8bea7, State: Running, Role: LEADER
I20260812 01:16:39.098482 22311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c0d57e12149768b50df0d78c8bea7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45823 } }
I20260812 01:16:39.099119 22300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:39.101095 22314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788c0d57e12149768b50df0d78c8bea7. Latest consensus state: current_term: 1 leader_uuid: "788c0d57e12149768b50df0d78c8bea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c0d57e12149768b50df0d78c8bea7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45823 } } }
I20260812 01:16:39.101487 22314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:39.102042 22312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788c0d57e12149768b50df0d78c8bea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788c0d57e12149768b50df0d78c8bea7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 45823 } } }
I20260812 01:16:39.102345 22312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:39.104272 22324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:39.108882 22324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:39.118700 22324 catalog_manager.cc:1347] Generated new cluster ID: 75d4bbc88daf4c608ef112f00cfc4a7d
I20260812 01:16:39.118837 22324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:39.142004 22324 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:39.143373 22324 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:39.149154 22324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7: Generated new TSK 0
I20260812 01:16:39.149936 22324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
Entry for principal kudu/127.20.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab.
I20260812 01:16:39.228207 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45823
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:39.589947 22349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:39.590267 22349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:39.590415 22349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:39.597965 22349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:39.598088 22349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:39.598260 22349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:16:39.608796 22349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.1.keytab
--principal=kudu/127.20.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:39.610203 22349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:39.611604 22349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:39.625224 22371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:39.625205 22361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:39.625589 22362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:39.625895 22349 server_base.cc:1034] running on GCE node
I20260812 01:16:39.626607 22349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:39.627663 22349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:39.628877 22349 hybrid_clock.cc:648] HybridClock initialized: now 1786497399628823 us; error 46 us; skew 500 ppm
I20260812 01:16:39.629216 22349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:39 dist-test-slave-21b9 krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497399, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:39.632458 22349 init.cc:377] Logged in from keytab as kudu/127.20.151.1@KRBTEST.COM (short username kudu)
I20260812 01:16:39.633947 22349 webserver.cc:460] Webserver started at http://127.20.151.1:35133/ using document root <none> and password file <none>
I20260812 01:16:39.634684 22349 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:39.634816 22349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:39.635161 22349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:39.638149 22349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "6fadf28e05264698addc4fba52a58c39"
format_stamp: "Formatted at 2026-08-12 01:16:39 on dist-test-slave-21b9"
server_key: "df59ba785bbfc7491c98b17b7db89794"
server_key_iv: "9e50aa82951d53aa25d274f992e0f02c"
server_key_version: "encryptionkey@0"
I20260812 01:16:39.638914 22349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "6fadf28e05264698addc4fba52a58c39"
format_stamp: "Formatted at 2026-08-12 01:16:39 on dist-test-slave-21b9"
server_key: "df59ba785bbfc7491c98b17b7db89794"
server_key_iv: "9e50aa82951d53aa25d274f992e0f02c"
server_key_version: "encryptionkey@0"
I20260812 01:16:39.644222 22349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 01:16:39.647729 22390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:39.649434 22349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 01:16:39.649626 22349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "6fadf28e05264698addc4fba52a58c39"
format_stamp: "Formatted at 2026-08-12 01:16:39 on dist-test-slave-21b9"
server_key: "df59ba785bbfc7491c98b17b7db89794"
server_key_iv: "9e50aa82951d53aa25d274f992e0f02c"
server_key_version: "encryptionkey@0"
I20260812 01:16:39.649824 22349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:39.666574 22349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:39.671129 22349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:39.671458 22349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:39.672510 22349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:39.673944 22349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:39.674041 22349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:39.674170 22349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:39.674245 22349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:39.706780 22349 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:43733
I20260812 01:16:39.706828 22583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:43733 every 8 connection(s)
I20260812 01:16:39.708303 22349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:16:39.715377 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 22349
I20260812 01:16:39.715564 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:16:39.716137 21084 external_mini_cluster.cc:1428] Setting key f57390527195ed6336b29b515792bdbe
Aug 12 01:16:39 dist-test-slave-21b9 krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497399, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.1@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:39.730587 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:39.710977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:50831 (local address 127.20.151.62:45823)
0812 01:16:39.711488 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:39.711493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:39.712470 (+ 977us) server_negotiation.cc:407] Connection header received
0812 01:16:39.714392 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:39.714399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:39.714485 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:39.714586 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:39.716682 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:39.717684 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:39.719991 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:39.720421 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:39.720580 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:39.724809 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:39.724816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:39.724820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:39.724856 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:39.727452 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:39.728287 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:39.728291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:39.728293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:39.728409 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:39.728862 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:39.728865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:39.728867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:39.729123 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:39.729244 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:39.730143 (+ 899us) server_negotiation.cc:299] Negotiation successful
0812 01:16:39.730327 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
I20260812 01:16:39.731932 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:39.711564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:45823 (local address 127.20.151.1:50831)
0812 01:16:39.712299 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:39.712339 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:39.714127 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:39.714847 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:39.714855 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:39.715645 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:39.716454 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:39.716483 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:39.717875 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:39.717879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:39.719824 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:39.719840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:39.720713 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:39.720721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:39.721010 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:39.721740 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:16:39.721769 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:39.724539 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:39.727592 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:39.727601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:39.727617 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:39.728178 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:39.728507 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:39.728519 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:39.728522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:39.728768 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:39.729254 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:39.729262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:39.729701 (+ 439us) client_negotiation.cc:769] Sending connection context
0812 01:16:39.730001 (+ 300us) client_negotiation.cc:240] Negotiation successful
0812 01:16:39.730550 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":288,"threads_started":1}
I20260812 01:16:39.733527 22585 heartbeater.cc:347] Connected to a master server at 127.20.151.62:45823
I20260812 01:16:39.733857 22585 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:39.734722 22585 heartbeater.cc:511] Master 127.20.151.62:45823 requested a full tablet report, sending...
I20260812 01:16:39.737195 22243 ts_manager.cc:194] Registered new tserver with Master: 6fadf28e05264698addc4fba52a58c39 (127.20.151.1:43733)
I20260812 01:16:39.739149 22243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.1@KRBTEST.COM'} at 127.20.151.1:50831
WARNING: no policy specified for kudu/127.20.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
Entry for principal kudu/127.20.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab.
I20260812 01:16:39.794636 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45823
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:40.157966 22593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:40.158262 22593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:40.158358 22593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:40.165607 22593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:40.165725 22593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:40.165843 22593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:16:40.175765 22593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.2.keytab
--principal=kudu/127.20.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:40.177052 22593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:40.178365 22593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:40.191293 22601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:40.191308 22603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:40.191684 22606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:40.192625 22593 server_base.cc:1034] running on GCE node
I20260812 01:16:40.193279 22593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:40.194238 22593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:40.195442 22593 hybrid_clock.cc:648] HybridClock initialized: now 1786497400195381 us; error 41 us; skew 500 ppm
I20260812 01:16:40.195767 22593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:40 dist-test-slave-21b9 krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497400, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:40.198803 22593 init.cc:377] Logged in from keytab as kudu/127.20.151.2@KRBTEST.COM (short username kudu)
I20260812 01:16:40.200366 22593 webserver.cc:460] Webserver started at http://127.20.151.2:46179/ using document root <none> and password file <none>
I20260812 01:16:40.201066 22593 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:40.201347 22593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:40.201689 22593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:40.204802 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "7cec984830cf4124959e24d03a456cce"
format_stamp: "Formatted at 2026-08-12 01:16:40 on dist-test-slave-21b9"
server_key: "ed0d583cc201114146122572c8c139ee"
server_key_iv: "715fb61fe13c2a46f1d84f84968553c5"
server_key_version: "encryptionkey@0"
I20260812 01:16:40.205562 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "7cec984830cf4124959e24d03a456cce"
format_stamp: "Formatted at 2026-08-12 01:16:40 on dist-test-slave-21b9"
server_key: "ed0d583cc201114146122572c8c139ee"
server_key_iv: "715fb61fe13c2a46f1d84f84968553c5"
server_key_version: "encryptionkey@0"
I20260812 01:16:40.211071 22593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:16:40.214641 22620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:40.216142 22593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:16:40.216295 22593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "7cec984830cf4124959e24d03a456cce"
format_stamp: "Formatted at 2026-08-12 01:16:40 on dist-test-slave-21b9"
server_key: "ed0d583cc201114146122572c8c139ee"
server_key_iv: "715fb61fe13c2a46f1d84f84968553c5"
server_key_version: "encryptionkey@0"
I20260812 01:16:40.216512 22593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:40.232322 22593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:40.236284 22593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:40.236553 22593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:40.237528 22593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:40.238893 22593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:40.238974 22593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:40.239105 22593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:40.239159 22593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:40.271780 22593 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:45389
I20260812 01:16:40.271831 22822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:45389 every 8 connection(s)
I20260812 01:16:40.273135 22593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:16:40.281975 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 22593
I20260812 01:16:40.282181 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:16:40.282644 21084 external_mini_cluster.cc:1428] Setting key c7277216e82b3b6b6c380f58e2eb13c4
Aug 12 01:16:40 dist-test-slave-21b9 krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497400, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.2@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:40.295787 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:40.275977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:40067 (local address 127.20.151.62:45823)
0812 01:16:40.276481 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:40.276486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:40.277504 (+ 1018us) server_negotiation.cc:407] Connection header received
0812 01:16:40.279427 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:40.279433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:40.279519 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:40.279619 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:40.281178 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.282109 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.284110 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.284723 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.284842 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:40.288991 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:40.288997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:40.289001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:40.289042 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:40.292093 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:40.293039 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:40.293044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:40.293046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:40.293187 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:40.293732 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:40.293736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:40.293739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:40.294081 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:40.294218 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:40.295248 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0812 01:16:40.295489 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":265,"threads_started":1}
I20260812 01:16:40.297210 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:40.276576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:45823 (local address 127.20.151.2:40067)
0812 01:16:40.277327 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:40.277370 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:40.279132 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:40.279873 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:40.279881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:40.280404 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:40.280996 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:40.281015 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.282319 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.282324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:40.283950 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:40.283970 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.284967 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.284974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:40.285223 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:40.286016 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:16:40.286045 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:40.288675 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:40.292264 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:40.292276 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:40.292296 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:40.292914 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:40.293295 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:40.293300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:40.293303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:40.293575 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:40.294221 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:40.294228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:40.294697 (+ 469us) client_negotiation.cc:769] Sending connection context
0812 01:16:40.295003 (+ 306us) client_negotiation.cc:240] Negotiation successful
0812 01:16:40.295772 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":295,"threads_started":1}
I20260812 01:16:40.298903 22823 heartbeater.cc:347] Connected to a master server at 127.20.151.62:45823
I20260812 01:16:40.299252 22823 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:40.300220 22823 heartbeater.cc:511] Master 127.20.151.62:45823 requested a full tablet report, sending...
I20260812 01:16:40.301956 22243 ts_manager.cc:194] Registered new tserver with Master: 7cec984830cf4124959e24d03a456cce (127.20.151.2:45389)
I20260812 01:16:40.303098 22243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.2@KRBTEST.COM'} at 127.20.151.2:40067
WARNING: no policy specified for kudu/127.20.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
Entry for principal kudu/127.20.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab.
I20260812 01:16:40.362136 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45823
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:40.714495 22831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:40.714749 22831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:40.714883 22831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:40.722338 22831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:40.722458 22831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:40.722615 22831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:16:40.732342 22831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.3.keytab
--principal=kudu/127.20.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:45823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:40.733661 22831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:40.735049 22831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 01:16:40.742357 22585 heartbeater.cc:503] Master 127.20.151.62:45823 was elected leader, sending a full tablet report...
W20260812 01:16:40.753834 22839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:40.753860 22840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:40.753805 22843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:40.754799 22831 server_base.cc:1034] running on GCE node
I20260812 01:16:40.755529 22831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:40.756558 22831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:40.757751 22831 hybrid_clock.cc:648] HybridClock initialized: now 1786497400757697 us; error 53 us; skew 500 ppm
I20260812 01:16:40.758061 22831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:40 dist-test-slave-21b9 krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497400, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:40.761106 22831 init.cc:377] Logged in from keytab as kudu/127.20.151.3@KRBTEST.COM (short username kudu)
I20260812 01:16:40.762602 22831 webserver.cc:460] Webserver started at http://127.20.151.3:45925/ using document root <none> and password file <none>
I20260812 01:16:40.763309 22831 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:40.763445 22831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:40.763760 22831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:40.766651 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "4bf7b25797934a72827ec5c34a9dc763"
format_stamp: "Formatted at 2026-08-12 01:16:40 on dist-test-slave-21b9"
server_key: "f66e74b522b1f41ae6acac9abfc4e9a0"
server_key_iv: "6852c77232df39233dea1ee6dae018ae"
server_key_version: "encryptionkey@0"
I20260812 01:16:40.767447 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "4bf7b25797934a72827ec5c34a9dc763"
format_stamp: "Formatted at 2026-08-12 01:16:40 on dist-test-slave-21b9"
server_key: "f66e74b522b1f41ae6acac9abfc4e9a0"
server_key_iv: "6852c77232df39233dea1ee6dae018ae"
server_key_version: "encryptionkey@0"
I20260812 01:16:40.772589 22831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 01:16:40.776255 22858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:40.777652 22831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:16:40.777838 22831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "4bf7b25797934a72827ec5c34a9dc763"
format_stamp: "Formatted at 2026-08-12 01:16:40 on dist-test-slave-21b9"
server_key: "f66e74b522b1f41ae6acac9abfc4e9a0"
server_key_iv: "6852c77232df39233dea1ee6dae018ae"
server_key_version: "encryptionkey@0"
I20260812 01:16:40.778053 22831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:40.815169 22831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:40.819687 22831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:40.819983 22831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:40.821098 22831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:40.822628 22831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:40.822722 22831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:40.822854 22831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:40.822911 22831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:40.855325 22831 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:46017
I20260812 01:16:40.855391 23053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:46017 every 8 connection(s)
I20260812 01:16:40.856642 22831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:16:40.859604 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 22831
I20260812 01:16:40.859820 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:16:40.860440 21084 external_mini_cluster.cc:1428] Setting key dc445e9f089bde30cc8686b095eec38a
Aug 12 01:16:40 dist-test-slave-21b9 krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497400, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.3@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:16:40.877430 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:40.859140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:51489 (local address 127.20.151.62:45823)
0812 01:16:40.859590 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:40.859595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:40.860806 (+ 1211us) server_negotiation.cc:407] Connection header received
0812 01:16:40.862994 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:40.863000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:40.863089 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:40.863196 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:40.864825 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.865635 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.867201 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.867761 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.867873 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:40.871356 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:40.871361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:40.871365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:40.871408 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:40.874230 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:40.875064 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:40.875068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:40.875071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:40.875285 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:40.875804 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:40.875810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:40.875813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:40.876135 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:40.876248 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:40.876958 (+ 710us) server_negotiation.cc:299] Negotiation successful
0812 01:16:40.877150 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":235,"threads_started":1}
I20260812 01:16:40.878388 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:40.859800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:45823 (local address 127.20.151.3:51489)
0812 01:16:40.860642 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:40.860689 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:40.862714 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:40.863455 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:40.863464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:40.864043 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:40.864652 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:40.864672 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.865773 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.865776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:40.867047 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:40.867062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.867955 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.867960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:40.868161 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:40.868806 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:16:40.868831 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:40.871094 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:40.874383 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:40.874393 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:40.874411 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:40.874940 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:40.875402 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:16:40.875407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:16:40.875410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:16:40.875695 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:16:40.876263 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:40.876270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:40.876627 (+ 357us) client_negotiation.cc:769] Sending connection context
0812 01:16:40.876871 (+ 244us) client_negotiation.cc:240] Negotiation successful
0812 01:16:40.877298 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":337,"threads_started":1}
I20260812 01:16:40.879750 23054 heartbeater.cc:347] Connected to a master server at 127.20.151.62:45823
I20260812 01:16:40.880040 23054 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:40.880755 23054 heartbeater.cc:511] Master 127.20.151.62:45823 requested a full tablet report, sending...
I20260812 01:16:40.882350 22243 ts_manager.cc:194] Registered new tserver with Master: 4bf7b25797934a72827ec5c34a9dc763 (127.20.151.3:46017)
I20260812 01:16:40.883219 22243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.151.3@KRBTEST.COM'} at 127.20.151.3:51489
I20260812 01:16:40.888222 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:16:40.905004 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:40.892626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.20.151.62:45823)
0812 01:16:40.892823 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:40.892828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:40.893246 (+ 418us) server_negotiation.cc:407] Connection header received
0812 01:16:40.893389 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:40.893392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:40.893459 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:40.893548 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:40.895004 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.895778 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.897056 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.897462 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.897554 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:40.900054 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:40.900059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:40.900063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:40.900096 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:40.902714 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:40.903367 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:40.903370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:40.903373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:40.903520 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:40.903814 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:40.903816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:40.903818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:40.904143 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:40.904325 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:40.904605 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 01:16:40.904778 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 01:16:40.909106 22243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:16:40.916182 22243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:16:40.950757 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:40.929033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43733 (local address 127.0.0.1:49476)
0812 01:16:40.941336 (+ 12303us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:40.941389 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:40.941602 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:40.944179 (+ 2577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:40.944184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:40.944208 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:40.944586 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:40.944596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.946281 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.946287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:40.947938 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:40.947952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.949719 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.949725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:40.949923 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:40.949977 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 01:16:40.950133 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 01:16:40.950333 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11977,"thread_start_us":11951,"threads_started":1}
I20260812 01:16:40.950918 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:40.929252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:45389 (local address 127.0.0.1:47500)
0812 01:16:40.942161 (+ 12909us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:40.942179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:40.942413 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:40.943490 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:40.943497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:40.943527 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:40.943997 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:40.944010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.945905 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.945910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:40.948283 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:40.948298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.950084 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.950091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:40.950286 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:40.950319 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 01:16:40.950393 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 01:16:40.950521 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12791,"mutex_wait_us":26,"thread_start_us":12668,"threads_started":1}
I20260812 01:16:40.951517 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:40.929419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.20.151.1:43733)
0812 01:16:40.943331 (+ 13912us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:40.943339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:40.943369 (+ 30us) server_negotiation.cc:407] Connection header received
0812 01:16:40.943468 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:40.943475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:40.943695 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:40.943872 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:40.944734 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.946150 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.948096 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.949499 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.949664 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:40.949908 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:40.950331 (+ 423us) server_negotiation.cc:299] Negotiation successful
0812 01:16:40.950517 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13719,"thread_start_us":13538,"threads_started":1}
I20260812 01:16:40.951823 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:40.929674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.20.151.2:45389)
0812 01:16:40.940458 (+ 10784us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:40.940469 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:40.942290 (+ 1821us) server_negotiation.cc:407] Connection header received
0812 01:16:40.942643 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:40.942653 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:40.942937 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:40.943116 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:40.944247 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.945758 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.948456 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.949857 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.950011 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:40.950211 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:40.951334 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0812 01:16:40.951517 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10572,"thread_start_us":10805,"threads_started":1}
I20260812 01:16:40.952172 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:40.942802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.20.151.3:46017)
0812 01:16:40.943667 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:40.943675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:40.944382 (+ 707us) server_negotiation.cc:407] Connection header received
0812 01:16:40.944543 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:40.944550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:40.944749 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:40.944943 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:40.945854 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.947433 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.949410 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.950722 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.950845 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:40.951075 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:40.951629 (+ 554us) server_negotiation.cc:299] Negotiation successful
0812 01:16:40.951792 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":536,"threads_started":1}
I20260812 01:16:40.952742 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:40.943655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:46017 (local address 127.0.0.1:46032)
0812 01:16:40.944253 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:40.944273 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:40.944395 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:40.945203 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:40.945208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:40.945234 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:40.945617 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:40.945627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.947644 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.947649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:40.949263 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:40.949276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:40.951226 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:40.951232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:40.951434 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:40.951465 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 01:16:40.952406 (+ 941us) client_negotiation.cc:240] Negotiation successful
0812 01:16:40.952509 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":317,"threads_started":1}
I20260812 01:16:40.955513 22474 tablet_service.cc:1467] Processing CreateTablet for tablet 994412d9f4904c1fb47a176287b20c57 (DEFAULT_TABLE table=test-table [id=25005f46447d433d9f1e6789ba39456d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:40.955852 22949 tablet_service.cc:1467] Processing CreateTablet for tablet 994412d9f4904c1fb47a176287b20c57 (DEFAULT_TABLE table=test-table [id=25005f46447d433d9f1e6789ba39456d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:40.957077 22949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994412d9f4904c1fb47a176287b20c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:40.957137 22474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994412d9f4904c1fb47a176287b20c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:40.957791 22725 tablet_service.cc:1467] Processing CreateTablet for tablet 994412d9f4904c1fb47a176287b20c57 (DEFAULT_TABLE table=test-table [id=25005f46447d433d9f1e6789ba39456d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:40.959162 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994412d9f4904c1fb47a176287b20c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:40.968819 23091 tablet_bootstrap.cc:492] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: Bootstrap starting.
I20260812 01:16:40.971861 23091 tablet_bootstrap.cc:654] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:40.973013 23091 log.cc:824] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:40.973403 23092 tablet_bootstrap.cc:492] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39: Bootstrap starting.
I20260812 01:16:40.976011 23093 tablet_bootstrap.cc:492] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: Bootstrap starting.
I20260812 01:16:40.977371 23092 tablet_bootstrap.cc:654] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:40.978547 23092 log.cc:824] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:40.979831 23091 tablet_bootstrap.cc:492] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: No bootstrap required, opened a new log
I20260812 01:16:40.980115 23093 tablet_bootstrap.cc:654] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:40.980130 23091 ts_tablet_manager.cc:1397] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260812 01:16:40.981292 23092 tablet_bootstrap.cc:492] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39: No bootstrap required, opened a new log
I20260812 01:16:40.981408 23093 log.cc:824] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:40.981539 23092 ts_tablet_manager.cc:1397] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260812 01:16:40.985922 23093 tablet_bootstrap.cc:492] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: No bootstrap required, opened a new log
I20260812 01:16:40.986202 23093 ts_tablet_manager.cc:1397] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 01:16:40.989431 23092 raft_consensus.cc:348] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadf28e05264698addc4fba52a58c39" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43733 } } peers { permanent_uuid: "7cec984830cf4124959e24d03a456cce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45389 } } peers { permanent_uuid: "4bf7b25797934a72827ec5c34a9dc763" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46017 } }
I20260812 01:16:40.989866 23092 raft_consensus.cc:374] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:40.989974 23092 raft_consensus.cc:729] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fadf28e05264698addc4fba52a58c39, State: Initialized, Role: FOLLOWER
I20260812 01:16:40.990582 23092 consensus_queue.cc:260] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadf28e05264698addc4fba52a58c39" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43733 } } peers { permanent_uuid: "7cec984830cf4124959e24d03a456cce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45389 } } peers { permanent_uuid: "4bf7b25797934a72827ec5c34a9dc763" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46017 } }
I20260812 01:16:40.991171 23091 raft_consensus.cc:348] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadf28e05264698addc4fba52a58c39" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43733 } } peers { permanent_uuid: "7cec984830cf4124959e24d03a456cce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45389 } } peers { permanent_uuid: "4bf7b25797934a72827ec5c34a9dc763" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46017 } }
I20260812 01:16:40.991626 23091 raft_consensus.cc:374] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:40.991715 23091 raft_consensus.cc:729] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf7b25797934a72827ec5c34a9dc763, State: Initialized, Role: FOLLOWER
I20260812 01:16:40.992362 23091 consensus_queue.cc:260] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadf28e05264698addc4fba52a58c39" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43733 } } peers { permanent_uuid: "7cec984830cf4124959e24d03a456cce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45389 } } peers { permanent_uuid: "4bf7b25797934a72827ec5c34a9dc763" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46017 } }
I20260812 01:16:40.997076 23093 raft_consensus.cc:348] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadf28e05264698addc4fba52a58c39" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43733 } } peers { permanent_uuid: "7cec984830cf4124959e24d03a456cce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45389 } } peers { permanent_uuid: "4bf7b25797934a72827ec5c34a9dc763" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46017 } }
I20260812 01:16:40.997622 23093 raft_consensus.cc:374] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:40.997726 23093 raft_consensus.cc:729] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cec984830cf4124959e24d03a456cce, State: Initialized, Role: FOLLOWER
I20260812 01:16:40.998486 23093 consensus_queue.cc:260] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadf28e05264698addc4fba52a58c39" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43733 } } peers { permanent_uuid: "7cec984830cf4124959e24d03a456cce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45389 } } peers { permanent_uuid: "4bf7b25797934a72827ec5c34a9dc763" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46017 } }
I20260812 01:16:40.998638 23054 heartbeater.cc:503] Master 127.20.151.62:45823 was elected leader, sending a full tablet report...
I20260812 01:16:40.999389 23091 ts_tablet_manager.cc:1428] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: Time spent starting tablet: real 0.019s user 0.011s sys 0.008s
I20260812 01:16:41.000512 23092 ts_tablet_manager.cc:1428] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39: Time spent starting tablet: real 0.019s user 0.018s sys 0.002s
I20260812 01:16:41.001257 22823 heartbeater.cc:503] Master 127.20.151.62:45823 was elected leader, sending a full tablet report...
I20260812 01:16:41.002771 23093 ts_tablet_manager.cc:1428] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260812 01:16:41.025148 22824 tablet.cc:2369] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:16:41.095507 23101 raft_consensus.cc:482] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:16:41.095850 23101 raft_consensus.cc:504] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadf28e05264698addc4fba52a58c39" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43733 } } peers { permanent_uuid: "7cec984830cf4124959e24d03a456cce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45389 } } peers { permanent_uuid: "4bf7b25797934a72827ec5c34a9dc763" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46017 } }
I20260812 01:16:41.097718 23101 leader_election.cc:288] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fadf28e05264698addc4fba52a58c39 (127.20.151.1:43733), 7cec984830cf4124959e24d03a456cce (127.20.151.2:45389)
I20260812 01:16:41.104251 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:41.098290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:41357 (local address 127.20.151.2:45389)
0812 01:16:41.098525 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:41.098531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:41.098552 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:16:41.098634 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:41.098639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:41.098716 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:41.098809 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:41.099331 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.100336 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.102204 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.103358 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.103472 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:41.103527 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:41.103923 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 01:16:41.104052 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 01:16:41.104262 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:41.098089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:45389 (local address 127.20.151.3:41357)
0812 01:16:41.098300 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:41.098320 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:41.098471 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:41.098869 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:41.098878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:41.098917 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:41.099137 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:41.099147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.100510 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.100516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:41.102046 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:41.102060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.103501 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.103508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:41.103699 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:41.103734 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 01:16:41.103839 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 01:16:41.103976 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 01:16:41.105247 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:41.098270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:53061 (local address 127.20.151.1:43733)
0812 01:16:41.098562 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:41.098570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:41.099210 (+ 640us) server_negotiation.cc:407] Connection header received
0812 01:16:41.099367 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:41.099370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:41.099433 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:41.099504 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:41.100098 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.101076 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.103121 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.104229 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.104398 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:41.104470 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:41.104909 (+ 439us) server_negotiation.cc:299] Negotiation successful
0812 01:16:41.105062 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260812 01:16:41.105249 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:41.098092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:43733 (local address 127.20.151.3:53061)
0812 01:16:41.099106 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:41.099120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:41.099221 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:41.099540 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:41.099545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:41.099570 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:41.099931 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:41.099943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.101240 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.101244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:41.102955 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:41.102972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.104544 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.104551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:41.104717 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:41.104750 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 01:16:41.104832 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 01:16:41.104923 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":183,"thread_start_us":432,"threads_started":1}
I20260812 01:16:41.105664 22753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994412d9f4904c1fb47a176287b20c57" candidate_uuid: "4bf7b25797934a72827ec5c34a9dc763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cec984830cf4124959e24d03a456cce" is_pre_election: true
I20260812 01:16:41.106273 22753 raft_consensus.cc:2454] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bf7b25797934a72827ec5c34a9dc763 in term 0.
I20260812 01:16:41.106343 22508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994412d9f4904c1fb47a176287b20c57" candidate_uuid: "4bf7b25797934a72827ec5c34a9dc763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fadf28e05264698addc4fba52a58c39" is_pre_election: true
I20260812 01:16:41.106912 22508 raft_consensus.cc:2454] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bf7b25797934a72827ec5c34a9dc763 in term 0.
I20260812 01:16:41.106995 22863 leader_election.cc:302] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf7b25797934a72827ec5c34a9dc763, 7cec984830cf4124959e24d03a456cce; no voters:
I20260812 01:16:41.107548 23101 raft_consensus.cc:2781] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:16:41.107714 23101 raft_consensus.cc:482] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:16:41.107818 23101 raft_consensus.cc:3037] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 0 FOLLOWER]: Advancing to term 1
W20260812 01:16:41.108184 23056 tablet.cc:2369] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:16:41.110232 23101 raft_consensus.cc:504] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadf28e05264698addc4fba52a58c39" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43733 } } peers { permanent_uuid: "7cec984830cf4124959e24d03a456cce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45389 } } peers { permanent_uuid: "4bf7b25797934a72827ec5c34a9dc763" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46017 } }
I20260812 01:16:41.110841 23101 leader_election.cc:288] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [CANDIDATE]: Term 1 election: Requested vote from peers 6fadf28e05264698addc4fba52a58c39 (127.20.151.1:43733), 7cec984830cf4124959e24d03a456cce (127.20.151.2:45389)
I20260812 01:16:41.111331 22508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994412d9f4904c1fb47a176287b20c57" candidate_uuid: "4bf7b25797934a72827ec5c34a9dc763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fadf28e05264698addc4fba52a58c39"
I20260812 01:16:41.111407 22753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994412d9f4904c1fb47a176287b20c57" candidate_uuid: "4bf7b25797934a72827ec5c34a9dc763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cec984830cf4124959e24d03a456cce"
I20260812 01:16:41.111609 22508 raft_consensus.cc:3037] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:41.111609 22753 raft_consensus.cc:3037] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:41.114756 22753 raft_consensus.cc:2454] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bf7b25797934a72827ec5c34a9dc763 in term 1.
I20260812 01:16:41.114756 22508 raft_consensus.cc:2454] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bf7b25797934a72827ec5c34a9dc763 in term 1.
I20260812 01:16:41.115175 22860 leader_election.cc:302] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf7b25797934a72827ec5c34a9dc763, 6fadf28e05264698addc4fba52a58c39; no voters:
I20260812 01:16:41.115512 23101 raft_consensus.cc:2781] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:41.116034 23101 raft_consensus.cc:686] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 1 LEADER]: Becoming Leader. State: Replica: 4bf7b25797934a72827ec5c34a9dc763, State: Running, Role: LEADER
I20260812 01:16:41.116591 23101 consensus_queue.cc:237] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadf28e05264698addc4fba52a58c39" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43733 } } peers { permanent_uuid: "7cec984830cf4124959e24d03a456cce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45389 } } peers { permanent_uuid: "4bf7b25797934a72827ec5c34a9dc763" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46017 } }
I20260812 01:16:41.121328 22243 catalog_manager.cc:5495] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bf7b25797934a72827ec5c34a9dc763 (127.20.151.3). New cstate: current_term: 1 leader_uuid: "4bf7b25797934a72827ec5c34a9dc763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadf28e05264698addc4fba52a58c39" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cec984830cf4124959e24d03a456cce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bf7b25797934a72827ec5c34a9dc763" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46017 } health_report { overall_health: HEALTHY } } }
I20260812 01:16:41.178646 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:41.173507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.20.151.3:46017)
0812 01:16:41.173725 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:41.173730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:41.173750 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:16:41.173859 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:41.173862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:41.173961 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:41.174104 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:16:41.174429 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.175205 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.176564 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.176976 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.177074 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:41.177563 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:16:41.178096 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:16:41.178218 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:41.178264 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 01:16:41.178346 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:16:41.187440 22508 raft_consensus.cc:1261] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39 [term 1 FOLLOWER]: Refusing update from remote peer 4bf7b25797934a72827ec5c34a9dc763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:41.187470 22753 raft_consensus.cc:1261] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce [term 1 FOLLOWER]: Refusing update from remote peer 4bf7b25797934a72827ec5c34a9dc763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:41.188547 23104 consensus_queue.cc:1035] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fadf28e05264698addc4fba52a58c39" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 43733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:16:41.188920 23101 consensus_queue.cc:1035] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cec984830cf4124959e24d03a456cce" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:16:41.201675 23108 mvcc.cc:205] Tried to move back new op lower bound from 7317493355252387840 to 7317493354977316864. Current Snapshot: MvccSnapshot[applied={T|T < 7317493355252387840}]
I20260812 01:16:41.215816 22243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:41.216112 22243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:41.220050 22243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 788c0d57e12149768b50df0d78c8bea7: Sending DeleteTablet for 3 replicas of tablet 994412d9f4904c1fb47a176287b20c57
I20260812 01:16:41.221261 22474 tablet_service.cc:1514] Processing DeleteTablet for tablet 994412d9f4904c1fb47a176287b20c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:41 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:49476
I20260812 01:16:41.221344 22949 tablet_service.cc:1514] Processing DeleteTablet for tablet 994412d9f4904c1fb47a176287b20c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:41 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:46032
I20260812 01:16:41.221453 22725 tablet_service.cc:1514] Processing DeleteTablet for tablet 994412d9f4904c1fb47a176287b20c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:41 UTC) from {username='kudu', principal='kudu/127.20.151.62@KRBTEST.COM'} at 127.0.0.1:47500
I20260812 01:16:41.222847 23138 tablet_replica.cc:330] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: stopping tablet replica
I20260812 01:16:41.223173 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 22349
I20260812 01:16:41.223412 23138 raft_consensus.cc:2229] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:16:41.224195 23138 raft_consensus.cc:2258] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:41.226518 23138 ts_tablet_manager.cc:1905] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:41.233610 23141 tablet_replica.cc:330] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: stopping tablet replica
I20260812 01:16:41.234025 23141 raft_consensus.cc:2229] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:41.234422 23138 ts_tablet_manager.cc:1918] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:41.234498 23141 raft_consensus.cc:2258] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:41.234630 23138 log.cc:1196] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/994412d9f4904c1fb47a176287b20c57
I20260812 01:16:41.235251 23138 ts_tablet_manager.cc:1939] T 994412d9f4904c1fb47a176287b20c57 P 4bf7b25797934a72827ec5c34a9dc763: Deleting consensus metadata
I20260812 01:16:41.238013 22227 catalog_manager.cc:4841] TS 4bf7b25797934a72827ec5c34a9dc763 (127.20.151.3:46017): tablet 994412d9f4904c1fb47a176287b20c57 (table test-table [id=25005f46447d433d9f1e6789ba39456d]) successfully deleted
I20260812 01:16:41.238406 23140 tablet_replica.cc:330] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39: stopping tablet replica
I20260812 01:16:41.238387 23141 ts_tablet_manager.cc:1905] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:41.239048 23140 raft_consensus.cc:2229] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:41.239647 23140 raft_consensus.cc:2258] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:41.241730 23140 ts_tablet_manager.cc:1905] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:41.245229 23141 ts_tablet_manager.cc:1918] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:41.245450 23141 log.cc:1196] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/994412d9f4904c1fb47a176287b20c57
I20260812 01:16:41.245988 23141 ts_tablet_manager.cc:1939] T 994412d9f4904c1fb47a176287b20c57 P 7cec984830cf4124959e24d03a456cce: Deleting consensus metadata
I20260812 01:16:41.247988 22227 catalog_manager.cc:4841] TS 7cec984830cf4124959e24d03a456cce (127.20.151.2:45389): tablet 994412d9f4904c1fb47a176287b20c57 (table test-table [id=25005f46447d433d9f1e6789ba39456d]) successfully deleted
I20260812 01:16:41.248433 23140 ts_tablet_manager.cc:1918] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:41.248590 23140 log.cc:1196] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/994412d9f4904c1fb47a176287b20c57
I20260812 01:16:41.249002 23140 ts_tablet_manager.cc:1939] T 994412d9f4904c1fb47a176287b20c57 P 6fadf28e05264698addc4fba52a58c39: Deleting consensus metadata
Aug 12 01:16:41 dist-test-slave-21b9 krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.1@KRBTEST.COM
I20260812 01:16:41.250417 22223 catalog_manager.cc:4841] TS 6fadf28e05264698addc4fba52a58c39 (127.20.151.1:43733): tablet 994412d9f4904c1fb47a176287b20c57 (table test-table [id=25005f46447d433d9f1e6789ba39456d]) successfully deleted
I20260812 01:16:41.256165 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:41.224611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.20.151.1:43733)
0812 01:16:41.224785 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:41.224791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:41.238062 (+ 13271us) server_negotiation.cc:407] Connection header received
0812 01:16:41.238635 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:41.238641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:41.238741 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:41.238843 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:41.240747 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.241933 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.243889 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.244531 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.244647 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:41.250397 (+ 5750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:41.250413 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:41.250431 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:41.250471 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:41.253252 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:41.254094 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:41.254101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:41.254104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:41.254236 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:41.254638 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:41.254641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:41.254643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:41.255037 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:41.255294 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:41.255725 (+ 431us) server_negotiation.cc:299] Negotiation successful
0812 01:16:41.255885 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:16:41.256763 22574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:41.382319 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 22349
I20260812 01:16:41.414168 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 22593
Aug 12 01:16:41 dist-test-slave-21b9 krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.2@KRBTEST.COM
I20260812 01:16:41.432473 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:41.415015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.20.151.2:45389)
0812 01:16:41.415179 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:41.415183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:41.415202 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:16:41.415402 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:41.415407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:41.415514 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:41.415613 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:41.417407 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.418245 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.420210 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.420620 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.420721 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:41.426143 (+ 5422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:41.426162 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:41.426179 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:41.426219 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:41.429469 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:41.430295 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:41.430303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:41.430305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:41.430433 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:41.430820 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:41.430823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:41.430824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:41.431212 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:41.431661 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:41.431993 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 01:16:41.432174 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 01:16:41.433132 22816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:41.523710 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 22593
I20260812 01:16:41.553128 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 22831
Aug 12 01:16:41 dist-test-slave-21b9 krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.3@KRBTEST.COM
I20260812 01:16:41.571218 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:41.553989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.20.151.3:46017)
0812 01:16:41.554150 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:41.554154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:41.554338 (+ 184us) server_negotiation.cc:407] Connection header received
0812 01:16:41.554495 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:41.554502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:41.554634 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:41.554753 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:41.556245 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.557460 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.558785 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:41.559510 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:41.559649 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:41.564074 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:41.564090 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:41.564118 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:41.564166 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:41.568231 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:41.568936 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:41.568947 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:41.568950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:41.569158 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:41.569550 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:41.569555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:41.569558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:41.570047 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:41.570393 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:41.570582 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 01:16:41.570814 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 01:16:41.572014 23047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:41.700070 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 22831
I20260812 01:16:41.731925 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 22178
I20260812 01:16:41.733004 22293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:41.845237 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 22178
2026-08-12T01:16:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4927 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T01:16:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:41Z Disabled control of system clock
I20260812 01:16:41.954309 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:39751
--webserver_interface=127.20.151.62
--webserver_port=0
--builtin_ntp_servers=127.20.151.20:46623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:39751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:16:42.311404 23161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:42.311699 23161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:42.311793 23161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:42.319120 23161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:42.319237 23161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:42.319303 23161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:42.319377 23161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:42.319424 23161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:42.328939 23161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:46623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:39751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:39751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:42.330305 23161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:42.332263 23161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:42.344190 23170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:42.344233 23176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:42.344156 23172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:42.345467 23161 server_base.cc:1034] running on GCE node
I20260812 01:16:42.346335 23161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:42.347891 23161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:42.349123 23161 hybrid_clock.cc:648] HybridClock initialized: now 1786497402349073 us; error 44 us; skew 500 ppm
I20260812 01:16:42.349545 23161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:42.352088 23161 webserver.cc:460] Webserver started at http://127.20.151.62:33435/ using document root <none> and password file <none>
I20260812 01:16:42.352830 23161 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:42.352942 23161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:42.353358 23161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:42.356554 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "bb5263c4677f40ba9fc805b127c33445"
format_stamp: "Formatted at 2026-08-12 01:16:42 on dist-test-slave-21b9"
server_key: "77a96e063547bfba44c00186876fa7f3"
server_key_iv: "342fe72e2d1df6ea6631b8cb49b2f98a"
server_key_version: "encryptionkey@0"
I20260812 01:16:42.357414 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "bb5263c4677f40ba9fc805b127c33445"
format_stamp: "Formatted at 2026-08-12 01:16:42 on dist-test-slave-21b9"
server_key: "77a96e063547bfba44c00186876fa7f3"
server_key_iv: "342fe72e2d1df6ea6631b8cb49b2f98a"
server_key_version: "encryptionkey@0"
I20260812 01:16:42.362874 23161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:16:42.366519 23185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:42.367957 23161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:16:42.368156 23161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "bb5263c4677f40ba9fc805b127c33445"
format_stamp: "Formatted at 2026-08-12 01:16:42 on dist-test-slave-21b9"
server_key: "77a96e063547bfba44c00186876fa7f3"
server_key_iv: "342fe72e2d1df6ea6631b8cb49b2f98a"
server_key_version: "encryptionkey@0"
I20260812 01:16:42.368371 23161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:42.383802 23161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:42.384666 23161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:42.384949 23161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:42.403000 23161 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:39751
I20260812 01:16:42.403021 23280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:39751 every 8 connection(s)
I20260812 01:16:42.404386 23161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:42.408216 23281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:42.410867 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 23161
I20260812 01:16:42.411048 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:42.411625 21084 external_mini_cluster.cc:1428] Setting key 5d83442c1f6d95906eea2bacad458dd9
I20260812 01:16:42.420527 23281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445: Bootstrap starting.
I20260812 01:16:42.423902 23281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:42.425096 23281 log.cc:824] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:42.425518 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:42.413725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.20.151.62:39751)
0812 01:16:42.414803 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:42.414814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:42.414857 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:16:42.416622 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:42.416647 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:42.417105 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:42.417476 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:42.418130 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:42.419562 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:42.421038 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:42.421711 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:42.421884 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:42.423160 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:42.423169 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:42.423183 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:42.423769 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:42.423875 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:42.423901 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:42.424018 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:42.424240 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 01:16:42.424555 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":371,"threads_started":1}
I20260812 01:16:42.428364 23281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445: No bootstrap required, opened a new log
I20260812 01:16:42.435885 23281 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5263c4677f40ba9fc805b127c33445" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39751 } }
I20260812 01:16:42.436291 23281 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:42.436389 23281 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5263c4677f40ba9fc805b127c33445, State: Initialized, Role: FOLLOWER
I20260812 01:16:42.436932 23281 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5263c4677f40ba9fc805b127c33445" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39751 } }
I20260812 01:16:42.437175 23281 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:42.437301 23281 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:42.437471 23281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:42.439915 23281 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5263c4677f40ba9fc805b127c33445" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39751 } }
I20260812 01:16:42.440404 23281 leader_election.cc:302] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb5263c4677f40ba9fc805b127c33445; no voters:
I20260812 01:16:42.440986 23281 leader_election.cc:288] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:42.441134 23290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:42.441872 23290 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [term 1 LEADER]: Becoming Leader. State: Replica: bb5263c4677f40ba9fc805b127c33445, State: Running, Role: LEADER
I20260812 01:16:42.442406 23290 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5263c4677f40ba9fc805b127c33445" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39751 } }
I20260812 01:16:42.443046 23281 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:42.445349 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb5263c4677f40ba9fc805b127c33445. Latest consensus state: current_term: 1 leader_uuid: "bb5263c4677f40ba9fc805b127c33445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5263c4677f40ba9fc805b127c33445" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39751 } } }
I20260812 01:16:42.445781 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:42.446589 23294 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb5263c4677f40ba9fc805b127c33445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5263c4677f40ba9fc805b127c33445" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 39751 } } }
I20260812 01:16:42.446985 23294 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:42.448441 23310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:42.453444 23310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:42.462225 23310 catalog_manager.cc:1347] Generated new cluster ID: c0d02dac5cbb49f4984d37e87b5322f9
I20260812 01:16:42.462385 23310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:42.479668 23310 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:42.480958 23310 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:42.492360 23310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445: Generated new TSK 0
I20260812 01:16:42.493265 23310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:16:42.501089 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.1:0
--local_ip_for_outbound_sockets=127.20.151.1
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39751
--builtin_ntp_servers=127.20.151.20:46623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:16:42.865149 23328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:42.865469 23328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:42.865562 23328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:42.873373 23328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:42.873489 23328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:42.873602 23328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.1
I20260812 01:16:42.882920 23328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:46623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.151.1
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.1
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:42.884271 23328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:42.886062 23328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:42.899588 23341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:42.899736 23335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:42.899830 23337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:42.900825 23328 server_base.cc:1034] running on GCE node
I20260812 01:16:42.901471 23328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:42.902475 23328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:42.903697 23328 hybrid_clock.cc:648] HybridClock initialized: now 1786497402903638 us; error 44 us; skew 500 ppm
I20260812 01:16:42.904040 23328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:42.906550 23328 webserver.cc:460] Webserver started at http://127.20.151.1:34371/ using document root <none> and password file <none>
I20260812 01:16:42.907320 23328 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:42.907420 23328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:42.907709 23328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:42.910650 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/data/instance:
uuid: "3c348f808d02485fb3d63d4dae03cfb1"
format_stamp: "Formatted at 2026-08-12 01:16:42 on dist-test-slave-21b9"
server_key: "b4bd7f0890b52d129233d50cc0349b93"
server_key_iv: "da8d372cf3c1f20268aa2541728ffdf9"
server_key_version: "encryptionkey@0"
I20260812 01:16:42.911477 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance:
uuid: "3c348f808d02485fb3d63d4dae03cfb1"
format_stamp: "Formatted at 2026-08-12 01:16:42 on dist-test-slave-21b9"
server_key: "b4bd7f0890b52d129233d50cc0349b93"
server_key_iv: "da8d372cf3c1f20268aa2541728ffdf9"
server_key_version: "encryptionkey@0"
I20260812 01:16:42.916555 23328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 01:16:42.920243 23352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:42.921576 23328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:16:42.921736 23328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/wal
uuid: "3c348f808d02485fb3d63d4dae03cfb1"
format_stamp: "Formatted at 2026-08-12 01:16:42 on dist-test-slave-21b9"
server_key: "b4bd7f0890b52d129233d50cc0349b93"
server_key_iv: "da8d372cf3c1f20268aa2541728ffdf9"
server_key_version: "encryptionkey@0"
I20260812 01:16:42.921941 23328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:42.951169 23328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:42.952174 23328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:42.952497 23328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:42.953599 23328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:42.955156 23328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:42.955259 23328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:42.955396 23328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:42.955492 23328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:42.988700 23328 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.1:42783
I20260812 01:16:42.988775 23560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.1:42783 every 8 connection(s)
I20260812 01:16:42.990023 23328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/data/info.pb
I20260812 01:16:42.998520 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 23328
I20260812 01:16:42.998709 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/wal/instance
I20260812 01:16:42.999132 21084 external_mini_cluster.cc:1428] Setting key 9e975522ba9f0738b819ff26ea1eb1b9
I20260812 01:16:43.002077 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.2:0
--local_ip_for_outbound_sockets=127.20.151.2
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39751
--builtin_ntp_servers=127.20.151.20:46623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:16:43.003182 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:42.992810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.1:33071 (local address 127.20.151.62:39751)
0812 01:16:42.993287 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:42.993294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:42.994622 (+ 1328us) server_negotiation.cc:407] Connection header received
0812 01:16:42.996487 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:42.996492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:42.996579 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:42.996673 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:42.997731 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:42.998440 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.000354 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.000854 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.000963 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:43.002384 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:43.002389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:43.002392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:43.002409 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:43.002467 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:43.002478 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:43.002593 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:43.002834 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 01:16:43.002915 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":234,"threads_started":1}
I20260812 01:16:43.004570 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:42.993421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39751 (local address 127.20.151.1:33071)
0812 01:16:42.994413 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:42.994461 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:42.996236 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:42.996903 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:42.996911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:42.996995 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:42.997582 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:42.997604 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:42.998600 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:42.998604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:43.000189 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:43.000205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.001096 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.001105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:43.001319 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:43.002031 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:16:43.002033 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:43.002080 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:16:43.002165 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:43.002610 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:43.002622 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:43.002631 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 01:16:43.002821 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 01:16:43.003215 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":299,"threads_started":1}
I20260812 01:16:43.006166 23562 heartbeater.cc:347] Connected to a master server at 127.20.151.62:39751
I20260812 01:16:43.006501 23562 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:43.007463 23562 heartbeater.cc:511] Master 127.20.151.62:39751 requested a full tablet report, sending...
I20260812 01:16:43.009657 23217 ts_manager.cc:194] Registered new tserver with Master: 3c348f808d02485fb3d63d4dae03cfb1 (127.20.151.1:42783)
I20260812 01:16:43.011147 23217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.1:33071
W20260812 01:16:43.352733 23566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:43.353024 23566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:43.353132 23566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:43.360448 23566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:43.360569 23566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:43.360709 23566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.2
I20260812 01:16:43.369915 23566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:46623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.151.2
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.2
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:43.371294 23566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:43.373163 23566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:43.386629 23577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:43.386601 23578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:43.386543 23580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:43.387456 23566 server_base.cc:1034] running on GCE node
I20260812 01:16:43.388123 23566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:43.389159 23566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:43.390381 23566 hybrid_clock.cc:648] HybridClock initialized: now 1786497403390331 us; error 41 us; skew 500 ppm
I20260812 01:16:43.390726 23566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:43.393067 23566 webserver.cc:460] Webserver started at http://127.20.151.2:46371/ using document root <none> and password file <none>
I20260812 01:16:43.393780 23566 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:43.393890 23566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:43.394224 23566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:43.397212 23566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/data/instance:
uuid: "172d884359514499ac55005cecacd353"
format_stamp: "Formatted at 2026-08-12 01:16:43 on dist-test-slave-21b9"
server_key: "cc89a7bd0e504b94a0d00362a97b71a4"
server_key_iv: "b04e0dc6f612f4efdec79c24016f6942"
server_key_version: "encryptionkey@0"
I20260812 01:16:43.398026 23566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance:
uuid: "172d884359514499ac55005cecacd353"
format_stamp: "Formatted at 2026-08-12 01:16:43 on dist-test-slave-21b9"
server_key: "cc89a7bd0e504b94a0d00362a97b71a4"
server_key_iv: "b04e0dc6f612f4efdec79c24016f6942"
server_key_version: "encryptionkey@0"
I20260812 01:16:43.403626 23566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 01:16:43.407958 23591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:43.409487 23566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:16:43.409644 23566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/wal
uuid: "172d884359514499ac55005cecacd353"
format_stamp: "Formatted at 2026-08-12 01:16:43 on dist-test-slave-21b9"
server_key: "cc89a7bd0e504b94a0d00362a97b71a4"
server_key_iv: "b04e0dc6f612f4efdec79c24016f6942"
server_key_version: "encryptionkey@0"
I20260812 01:16:43.409874 23566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:43.436651 23566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:43.437462 23566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:43.437733 23566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:43.438696 23566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:43.440222 23566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:43.440311 23566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:43.440394 23566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:43.440457 23566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:43.471959 23566 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.2:45535
I20260812 01:16:43.472047 23788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.2:45535 every 8 connection(s)
I20260812 01:16:43.473304 23566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/data/info.pb
I20260812 01:16:43.479115 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 23566
I20260812 01:16:43.479323 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/wal/instance
I20260812 01:16:43.479933 21084 external_mini_cluster.cc:1428] Setting key e6a38d97247a61be8afa294883515b8e
I20260812 01:16:43.482705 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.151.3:0
--local_ip_for_outbound_sockets=127.20.151.3
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39751
--builtin_ntp_servers=127.20.151.20:46623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:16:43.486078 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:43.475950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:46593 (local address 127.20.151.62:39751)
0812 01:16:43.476152 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:43.476159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:43.477524 (+ 1365us) server_negotiation.cc:407] Connection header received
0812 01:16:43.479453 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:43.479460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:43.479560 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:43.479676 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:43.480758 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.481578 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.483379 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.483957 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.484082 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:43.485359 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:43.485365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:43.485369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:43.485390 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:43.485430 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:43.485444 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:43.485524 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:43.485701 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 01:16:43.485796 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 01:16:43.487046 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:43.476405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39751 (local address 127.20.151.2:46593)
0812 01:16:43.477365 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:43.477408 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:43.479099 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:43.479891 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:43.479899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:43.480015 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:43.480609 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:43.480627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.481771 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.481776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:43.483223 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:43.483237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.484161 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.484165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:43.484370 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:43.485039 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:16:43.485042 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:43.485083 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:16:43.485170 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:43.485531 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:43.485537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:43.485543 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 01:16:43.485678 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 01:16:43.485976 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":293,"threads_started":1}
I20260812 01:16:43.488552 23789 heartbeater.cc:347] Connected to a master server at 127.20.151.62:39751
I20260812 01:16:43.488881 23789 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:43.489786 23789 heartbeater.cc:511] Master 127.20.151.62:39751 requested a full tablet report, sending...
I20260812 01:16:43.491437 23217 ts_manager.cc:194] Registered new tserver with Master: 172d884359514499ac55005cecacd353 (127.20.151.2:45535)
I20260812 01:16:43.492291 23217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.2:46593
W20260812 01:16:43.833865 23797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:43.834160 23797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:43.834265 23797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:43.841621 23797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:43.841750 23797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:43.841863 23797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.151.3
I20260812 01:16:43.851413 23797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:46623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.151.3
--webserver_port=0
--tserver_master_addrs=127.20.151.62:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.151.3
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:43.852720 23797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:43.854518 23797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:43.867465 23805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:43.867499 23806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:43.867545 23809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:43.868382 23797 server_base.cc:1034] running on GCE node
I20260812 01:16:43.868980 23797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:43.869923 23797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:43.871142 23797 hybrid_clock.cc:648] HybridClock initialized: now 1786497403871078 us; error 50 us; skew 500 ppm
I20260812 01:16:43.871485 23797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:16:43.873943 23797 webserver.cc:460] Webserver started at http://127.20.151.3:45531/ using document root <none> and password file <none>
I20260812 01:16:43.874634 23797 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:43.874727 23797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:43.874989 23797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:43.878019 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/data/instance:
uuid: "53592d1590644865beab8ccfba2074fa"
format_stamp: "Formatted at 2026-08-12 01:16:43 on dist-test-slave-21b9"
server_key: "4d4e67a49071f6acbd26ce19f96781e3"
server_key_iv: "5371edee77caa76547355e14ec5cb699"
server_key_version: "encryptionkey@0"
I20260812 01:16:43.878780 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance:
uuid: "53592d1590644865beab8ccfba2074fa"
format_stamp: "Formatted at 2026-08-12 01:16:43 on dist-test-slave-21b9"
server_key: "4d4e67a49071f6acbd26ce19f96781e3"
server_key_iv: "5371edee77caa76547355e14ec5cb699"
server_key_version: "encryptionkey@0"
I20260812 01:16:43.883737 23797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260812 01:16:43.887137 23819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:43.888515 23797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:16:43.888671 23797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/wal
uuid: "53592d1590644865beab8ccfba2074fa"
format_stamp: "Formatted at 2026-08-12 01:16:43 on dist-test-slave-21b9"
server_key: "4d4e67a49071f6acbd26ce19f96781e3"
server_key_iv: "5371edee77caa76547355e14ec5cb699"
server_key_version: "encryptionkey@0"
I20260812 01:16:43.888866 23797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:43.908588 23797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:43.909471 23797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:43.909772 23797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:43.910794 23797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:16:43.912350 23797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:16:43.912451 23797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:43.912564 23797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:16:43.912639 23797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:43.944710 23797 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.3:46815
I20260812 01:16:43.944803 24030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.3:46815 every 8 connection(s)
I20260812 01:16:43.946444 23797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/data/info.pb
I20260812 01:16:43.953106 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 23797
I20260812 01:16:43.953318 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/wal/instance
I20260812 01:16:43.953958 21084 external_mini_cluster.cc:1428] Setting key 67644d8eba5bdc86970ce433d34dabc9
I20260812 01:16:43.962353 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:43.949385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.3:45177 (local address 127.20.151.62:39751)
0812 01:16:43.949661 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:43.949667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:43.953812 (+ 4145us) server_negotiation.cc:407] Connection header received
0812 01:16:43.956220 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:43.956225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:43.956308 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:43.956410 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:43.957496 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.958276 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.959910 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.960320 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.960415 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:43.961697 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:43.961702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:43.961705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:43.961723 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:43.961754 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:43.961765 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:43.961833 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:43.961997 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 01:16:43.962082 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:16:43.963366 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:43.950174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:39751 (local address 127.20.151.3:45177)
0812 01:16:43.953599 (+ 3425us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:43.953661 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:43.955944 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:43.956586 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:43.956595 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:43.956693 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:16:43.957343 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:43.957361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.958414 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.958418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:43.959768 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:43.959781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.960489 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.960493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:43.960713 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:43.961389 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:16:43.961392 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:16:43.961434 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:16:43.961528 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:16:43.961838 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:16:43.961843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:16:43.961848 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 01:16:43.961984 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 01:16:43.962289 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2798,"thread_start_us":2576,"threads_started":1}
I20260812 01:16:43.964712 24031 heartbeater.cc:347] Connected to a master server at 127.20.151.62:39751
I20260812 01:16:43.964982 24031 heartbeater.cc:464] Registering TS with master...
I20260812 01:16:43.965718 24031 heartbeater.cc:511] Master 127.20.151.62:39751 requested a full tablet report, sending...
I20260812 01:16:43.967049 23217 ts_manager.cc:194] Registered new tserver with Master: 53592d1590644865beab8ccfba2074fa (127.20.151.3:46815)
I20260812 01:16:43.968092 23217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.151.3:45177
I20260812 01:16:43.969465 21084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:16:43.979687 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:43.973966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55892 (local address 127.20.151.62:39751)
0812 01:16:43.974135 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:43.974140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:43.974647 (+ 507us) server_negotiation.cc:407] Connection header received
0812 01:16:43.974821 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:43.974825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:43.974899 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:43.974978 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:43.975533 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.976212 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.977498 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:43.977911 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:43.978018 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:43.978990 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:43.978995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:43.978998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:43.979015 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:43.979055 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:43.979066 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:43.979139 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:43.979290 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 01:16:43.979386 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 01:16:43.983371 23217 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:16:43.991029 23217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:16:44.014686 23562 heartbeater.cc:503] Master 127.20.151.62:39751 was elected leader, sending a full tablet report...
I20260812 01:16:44.040729 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.004972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.20.151.1:42783)
0812 01:16:44.028967 (+ 23995us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:44.028991 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:44.031562 (+ 2571us) server_negotiation.cc:407] Connection header received
0812 01:16:44.031670 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:44.031677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:44.031919 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:44.032222 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:44.033342 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.034871 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.038002 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.039196 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.039374 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.039662 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:44.040184 (+ 522us) server_negotiation.cc:299] Negotiation successful
0812 01:16:44.040365 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23743,"thread_start_us":24031,"threads_started":1}
I20260812 01:16:44.040730 24050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.004232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:42783 (local address 127.0.0.1:36540)
0812 01:16:44.031295 (+ 27063us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:44.031312 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:44.031423 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:44.032577 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:44.032585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:44.032613 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:44.033103 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:44.033116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.035285 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.035291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:44.037750 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:44.037766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.039824 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.039830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:44.039972 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.039995 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 01:16:44.040107 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 01:16:44.040329 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26928,"mutex_wait_us":108,"thread_start_us":26371,"threads_started":1}
I20260812 01:16:44.040772 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.003871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46376 (local address 127.20.151.3:46815)
0812 01:16:44.032720 (+ 28849us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:44.032728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:44.032763 (+ 35us) server_negotiation.cc:407] Connection header received
0812 01:16:44.032854 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:44.032861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:44.033116 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:44.033821 (+ 705us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:44.034086 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.035299 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.037976 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.039335 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.039456 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.039696 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:44.040266 (+ 570us) server_negotiation.cc:299] Negotiation successful
0812 01:16:44.040420 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28652,"thread_start_us":28614,"threads_started":1}
I20260812 01:16:44.040769 24055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.004459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.20.151.2:45535)
0812 01:16:44.030704 (+ 26245us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:44.030713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:44.031040 (+ 327us) server_negotiation.cc:407] Connection header received
0812 01:16:44.031230 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:44.031240 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:44.031607 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:44.031845 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:44.032968 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.034610 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.037666 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.039081 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.039229 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.039551 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:44.040206 (+ 655us) server_negotiation.cc:299] Negotiation successful
0812 01:16:44.040386 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26058,"thread_start_us":26243,"threads_started":1}
I20260812 01:16:44.041486 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.003815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.2:45535 (local address 127.0.0.1:54906)
0812 01:16:44.030870 (+ 27055us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:44.030890 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:44.031052 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:44.032218 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:44.032226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:44.032258 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:44.032735 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:44.032750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.034997 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.035016 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:44.037471 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:44.037485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.039683 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.039689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:44.039925 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.039982 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 01:16:44.041032 (+ 1050us) client_negotiation.cc:240] Negotiation successful
0812 01:16:44.041149 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26908,"mutex_wait_us":553,"thread_start_us":25665,"threads_started":1}
I20260812 01:16:44.041623 24051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.003584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:46815 (local address 127.0.0.1:46376)
0812 01:16:44.029925 (+ 26341us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:44.029975 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:44.030219 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:44.033359 (+ 3140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:44.033364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:44.033389 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:44.033692 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:44.033701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.035463 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.035472 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:44.037750 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:44.037766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.039826 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.039833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:44.040034 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.040060 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 01:16:44.040130 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 01:16:44.041077 (+ 947us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26057,"mutex_wait_us":765,"thread_start_us":27695,"threads_started":1}
I20260812 01:16:44.045377 23678 tablet_service.cc:1467] Processing CreateTablet for tablet ee08e8b779bd4641a869ce993c241c3d (DEFAULT_TABLE table=test-table [id=e6d09c4b07c64382a197ba4ff1bdcff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:44.045409 23435 tablet_service.cc:1467] Processing CreateTablet for tablet ee08e8b779bd4641a869ce993c241c3d (DEFAULT_TABLE table=test-table [id=e6d09c4b07c64382a197ba4ff1bdcff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:44.045409 23910 tablet_service.cc:1467] Processing CreateTablet for tablet ee08e8b779bd4641a869ce993c241c3d (DEFAULT_TABLE table=test-table [id=e6d09c4b07c64382a197ba4ff1bdcff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:16:44.046732 23678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee08e8b779bd4641a869ce993c241c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:44.046898 23910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee08e8b779bd4641a869ce993c241c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:44.046896 23435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee08e8b779bd4641a869ce993c241c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:44.058831 24059 tablet_bootstrap.cc:492] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: Bootstrap starting.
I20260812 01:16:44.063508 24059 tablet_bootstrap.cc:654] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:44.063972 24071 tablet_bootstrap.cc:492] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: Bootstrap starting.
I20260812 01:16:44.064934 24059 log.cc:824] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:44.066509 24068 tablet_bootstrap.cc:492] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: Bootstrap starting.
I20260812 01:16:44.068005 24071 tablet_bootstrap.cc:654] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:44.068382 24059 tablet_bootstrap.cc:492] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: No bootstrap required, opened a new log
I20260812 01:16:44.068651 24059 ts_tablet_manager.cc:1397] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260812 01:16:44.069326 24071 log.cc:824] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:44.070303 24068 tablet_bootstrap.cc:654] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:44.071638 24068 log.cc:824] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:44.072865 24071 tablet_bootstrap.cc:492] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: No bootstrap required, opened a new log
I20260812 01:16:44.073148 24071 ts_tablet_manager.cc:1397] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260812 01:16:44.074472 24068 tablet_bootstrap.cc:492] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: No bootstrap required, opened a new log
I20260812 01:16:44.074728 24068 ts_tablet_manager.cc:1397] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260812 01:16:44.078858 24059 raft_consensus.cc:348] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592d1590644865beab8ccfba2074fa" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46815 } } peers { permanent_uuid: "172d884359514499ac55005cecacd353" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45535 } } peers { permanent_uuid: "3c348f808d02485fb3d63d4dae03cfb1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 42783 } }
I20260812 01:16:44.079375 24059 raft_consensus.cc:374] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:44.079483 24059 raft_consensus.cc:729] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 172d884359514499ac55005cecacd353, State: Initialized, Role: FOLLOWER
I20260812 01:16:44.080205 24059 consensus_queue.cc:260] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592d1590644865beab8ccfba2074fa" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46815 } } peers { permanent_uuid: "172d884359514499ac55005cecacd353" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45535 } } peers { permanent_uuid: "3c348f808d02485fb3d63d4dae03cfb1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 42783 } }
I20260812 01:16:44.081441 23789 heartbeater.cc:503] Master 127.20.151.62:39751 was elected leader, sending a full tablet report...
I20260812 01:16:44.082130 24059 ts_tablet_manager.cc:1428] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: Time spent starting tablet: real 0.013s user 0.006s sys 0.008s
I20260812 01:16:44.082404 24068 raft_consensus.cc:348] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592d1590644865beab8ccfba2074fa" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46815 } } peers { permanent_uuid: "172d884359514499ac55005cecacd353" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45535 } } peers { permanent_uuid: "3c348f808d02485fb3d63d4dae03cfb1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 42783 } }
I20260812 01:16:44.082846 24068 raft_consensus.cc:374] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:44.082942 24068 raft_consensus.cc:729] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c348f808d02485fb3d63d4dae03cfb1, State: Initialized, Role: FOLLOWER
I20260812 01:16:44.083297 24071 raft_consensus.cc:348] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592d1590644865beab8ccfba2074fa" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46815 } } peers { permanent_uuid: "172d884359514499ac55005cecacd353" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45535 } } peers { permanent_uuid: "3c348f808d02485fb3d63d4dae03cfb1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 42783 } }
I20260812 01:16:44.083745 24071 raft_consensus.cc:374] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:44.083837 24071 raft_consensus.cc:729] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53592d1590644865beab8ccfba2074fa, State: Initialized, Role: FOLLOWER
I20260812 01:16:44.083693 24068 consensus_queue.cc:260] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592d1590644865beab8ccfba2074fa" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46815 } } peers { permanent_uuid: "172d884359514499ac55005cecacd353" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45535 } } peers { permanent_uuid: "3c348f808d02485fb3d63d4dae03cfb1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 42783 } }
I20260812 01:16:44.084518 24071 consensus_queue.cc:260] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592d1590644865beab8ccfba2074fa" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46815 } } peers { permanent_uuid: "172d884359514499ac55005cecacd353" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45535 } } peers { permanent_uuid: "3c348f808d02485fb3d63d4dae03cfb1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 42783 } }
I20260812 01:16:44.086062 24068 ts_tablet_manager.cc:1428] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260812 01:16:44.087921 24031 heartbeater.cc:503] Master 127.20.151.62:39751 was elected leader, sending a full tablet report...
I20260812 01:16:44.088686 24071 ts_tablet_manager.cc:1428] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260812 01:16:44.198356 24033 tablet.cc:2369] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:16:44.225284 23792 tablet.cc:2369] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:16:44.242558 23563 tablet.cc:2369] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:16:44.320374 24075 raft_consensus.cc:482] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:16:44.320789 24075 raft_consensus.cc:504] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592d1590644865beab8ccfba2074fa" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46815 } } peers { permanent_uuid: "172d884359514499ac55005cecacd353" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45535 } } peers { permanent_uuid: "3c348f808d02485fb3d63d4dae03cfb1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 42783 } }
I20260812 01:16:44.322659 24075 leader_election.cc:288] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53592d1590644865beab8ccfba2074fa (127.20.151.3:46815), 3c348f808d02485fb3d63d4dae03cfb1 (127.20.151.1:42783)
I20260812 01:16:44.329566 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.323178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:45431 (local address 127.20.151.3:46815)
0812 01:16:44.323472 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:44.323480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:44.324061 (+ 581us) server_negotiation.cc:407] Connection header received
0812 01:16:44.324264 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:44.324269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:44.324330 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:44.324405 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:44.324963 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.325675 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.327578 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.328683 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.328846 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.328926 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:44.329261 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 01:16:44.329360 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 01:16:44.329739 24082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.323139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.3:46815 (local address 127.20.151.2:45431)
0812 01:16:44.323943 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:44.323960 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:44.324094 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:44.324456 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:44.324461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:44.324489 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:44.324820 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:44.324831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.325810 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.325816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:44.327377 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:44.327390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.328875 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.328881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:44.329067 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.329103 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 01:16:44.329215 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 01:16:44.329355 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":1,"thread_start_us":481,"threads_started":1}
I20260812 01:16:44.330596 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.323350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.151.2:43697 (local address 127.20.151.1:42783)
0812 01:16:44.323684 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:44.323691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:44.324662 (+ 971us) server_negotiation.cc:407] Connection header received
0812 01:16:44.324859 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:44.324862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:44.324944 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:44.325021 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:16:44.325605 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.326363 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.328430 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.329670 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.329828 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.329898 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:44.330292 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 01:16:44.330413 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260812 01:16:44.330621 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.323220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.1:42783 (local address 127.20.151.2:43697)
0812 01:16:44.324556 (+ 1336us) negotiation.cc:107] Waiting for socket to connect
0812 01:16:44.324573 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:16:44.324722 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:16:44.325062 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:16:44.325066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:16:44.325091 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:16:44.325420 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:44.325430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.326502 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.326506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:44.328243 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:16:44.328259 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.329941 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.329949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:16:44.330122 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.330151 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 01:16:44.330225 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 01:16:44.330331 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"thread_start_us":1000,"threads_started":1}
I20260812 01:16:44.330878 23951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08e8b779bd4641a869ce993c241c3d" candidate_uuid: "172d884359514499ac55005cecacd353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53592d1590644865beab8ccfba2074fa" is_pre_election: true
I20260812 01:16:44.331538 23951 raft_consensus.cc:2454] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 172d884359514499ac55005cecacd353 in term 0.
I20260812 01:16:44.331739 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08e8b779bd4641a869ce993c241c3d" candidate_uuid: "172d884359514499ac55005cecacd353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c348f808d02485fb3d63d4dae03cfb1" is_pre_election: true
I20260812 01:16:44.332290 23469 raft_consensus.cc:2454] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 172d884359514499ac55005cecacd353 in term 0.
I20260812 01:16:44.332391 23597 leader_election.cc:302] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 172d884359514499ac55005cecacd353, 53592d1590644865beab8ccfba2074fa; no voters:
I20260812 01:16:44.332839 24075 raft_consensus.cc:2781] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:16:44.333001 24075 raft_consensus.cc:482] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:16:44.333103 24075 raft_consensus.cc:3037] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:44.335541 24075 raft_consensus.cc:504] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592d1590644865beab8ccfba2074fa" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46815 } } peers { permanent_uuid: "172d884359514499ac55005cecacd353" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45535 } } peers { permanent_uuid: "3c348f808d02485fb3d63d4dae03cfb1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 42783 } }
I20260812 01:16:44.336128 24075 leader_election.cc:288] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [CANDIDATE]: Term 1 election: Requested vote from peers 53592d1590644865beab8ccfba2074fa (127.20.151.3:46815), 3c348f808d02485fb3d63d4dae03cfb1 (127.20.151.1:42783)
I20260812 01:16:44.336555 23951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08e8b779bd4641a869ce993c241c3d" candidate_uuid: "172d884359514499ac55005cecacd353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53592d1590644865beab8ccfba2074fa"
I20260812 01:16:44.336575 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08e8b779bd4641a869ce993c241c3d" candidate_uuid: "172d884359514499ac55005cecacd353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c348f808d02485fb3d63d4dae03cfb1"
I20260812 01:16:44.336742 23951 raft_consensus.cc:3037] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:44.336745 23469 raft_consensus.cc:3037] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:44.338902 23951 raft_consensus.cc:2454] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 172d884359514499ac55005cecacd353 in term 1.
I20260812 01:16:44.339012 23469 raft_consensus.cc:2454] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 172d884359514499ac55005cecacd353 in term 1.
I20260812 01:16:44.339363 23597 leader_election.cc:302] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 172d884359514499ac55005cecacd353, 53592d1590644865beab8ccfba2074fa; no voters:
I20260812 01:16:44.339687 24075 raft_consensus.cc:2781] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:44.340127 24075 raft_consensus.cc:686] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 1 LEADER]: Becoming Leader. State: Replica: 172d884359514499ac55005cecacd353, State: Running, Role: LEADER
I20260812 01:16:44.340654 24075 consensus_queue.cc:237] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592d1590644865beab8ccfba2074fa" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46815 } } peers { permanent_uuid: "172d884359514499ac55005cecacd353" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45535 } } peers { permanent_uuid: "3c348f808d02485fb3d63d4dae03cfb1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 42783 } }
I20260812 01:16:44.345271 23217 catalog_manager.cc:5495] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 reported cstate change: term changed from 0 to 1, leader changed from <none> to 172d884359514499ac55005cecacd353 (127.20.151.2). New cstate: current_term: 1 leader_uuid: "172d884359514499ac55005cecacd353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592d1590644865beab8ccfba2074fa" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "172d884359514499ac55005cecacd353" member_type: VOTER last_known_addr { host: "127.20.151.2" port: 45535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c348f808d02485fb3d63d4dae03cfb1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20260812 01:16:44.411952 24055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.406703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.20.151.2:45535)
0812 01:16:44.406916 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:44.406921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:44.406943 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:16:44.407050 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:44.407054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:44.407149 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:44.407316 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:16:44.407714 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.408427 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.409809 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.410222 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.410330 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.410810 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:16:44.411358 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:16:44.411494 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:44.411613 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 01:16:44.411691 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 01:16:44.421057 23469 raft_consensus.cc:1261] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1 [term 1 FOLLOWER]: Refusing update from remote peer 172d884359514499ac55005cecacd353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:44.421054 23951 raft_consensus.cc:1261] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa [term 1 FOLLOWER]: Refusing update from remote peer 172d884359514499ac55005cecacd353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:16:44.422037 24085 consensus_queue.cc:1035] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c348f808d02485fb3d63d4dae03cfb1" member_type: VOTER last_known_addr { host: "127.20.151.1" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:16:44.422361 24075 consensus_queue.cc:1035] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53592d1590644865beab8ccfba2074fa" member_type: VOTER last_known_addr { host: "127.20.151.3" port: 46815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:16:44.433235 24088 mvcc.cc:205] Tried to move back new op lower bound from 7317493368499089408 to 7317493368183164928. Current Snapshot: MvccSnapshot[applied={T|T < 7317493368499089408}]
I20260812 01:16:44.436723 24096 mvcc.cc:205] Tried to move back new op lower bound from 7317493368499089408 to 7317493368183164928. Current Snapshot: MvccSnapshot[applied={T|T < 7317493368499089408}]
I20260812 01:16:44.446427 23217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:44.446678 23217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:16:44.450343 23217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb5263c4677f40ba9fc805b127c33445: Sending DeleteTablet for 3 replicas of tablet ee08e8b779bd4641a869ce993c241c3d
I20260812 01:16:44.451587 23678 tablet_service.cc:1514] Processing DeleteTablet for tablet ee08e8b779bd4641a869ce993c241c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:44 UTC) from {username='slave'} at 127.0.0.1:54906
I20260812 01:16:44.451565 23910 tablet_service.cc:1514] Processing DeleteTablet for tablet ee08e8b779bd4641a869ce993c241c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:44 UTC) from {username='slave'} at 127.0.0.1:46376
I20260812 01:16:44.451719 23435 tablet_service.cc:1514] Processing DeleteTablet for tablet ee08e8b779bd4641a869ce993c241c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:16:44 UTC) from {username='slave'} at 127.0.0.1:36540
I20260812 01:16:44.456995 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 23328
I20260812 01:16:44.462118 24117 tablet_replica.cc:330] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: stopping tablet replica
I20260812 01:16:44.462625 24117 raft_consensus.cc:2229] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:44.463198 24117 raft_consensus.cc:2258] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:44.464402 24116 tablet_replica.cc:330] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: stopping tablet replica
I20260812 01:16:44.464830 24116 raft_consensus.cc:2229] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:16:44.465559 24116 raft_consensus.cc:2258] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:44.467455 24117 ts_tablet_manager.cc:1905] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:44.467608 24116 ts_tablet_manager.cc:1905] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:44.472383 24115 tablet_replica.cc:330] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: stopping tablet replica
I20260812 01:16:44.472888 24115 raft_consensus.cc:2229] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:16:44.473599 24115 raft_consensus.cc:2258] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:16:44.476467 24116 ts_tablet_manager.cc:1918] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:44.476722 24116 log.cc:1196] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-1/wal/wals/ee08e8b779bd4641a869ce993c241c3d
I20260812 01:16:44.476936 24117 ts_tablet_manager.cc:1918] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:44.477152 24115 ts_tablet_manager.cc:1905] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:16:44.477207 24117 log.cc:1196] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-0/wal/wals/ee08e8b779bd4641a869ce993c241c3d
I20260812 01:16:44.477360 24116 ts_tablet_manager.cc:1939] T ee08e8b779bd4641a869ce993c241c3d P 172d884359514499ac55005cecacd353: Deleting consensus metadata
I20260812 01:16:44.477710 24117 ts_tablet_manager.cc:1939] T ee08e8b779bd4641a869ce993c241c3d P 3c348f808d02485fb3d63d4dae03cfb1: Deleting consensus metadata
I20260812 01:16:44.479295 23194 catalog_manager.cc:4841] TS 172d884359514499ac55005cecacd353 (127.20.151.2:45535): tablet ee08e8b779bd4641a869ce993c241c3d (table test-table [id=e6d09c4b07c64382a197ba4ff1bdcff1]) successfully deleted
I20260812 01:16:44.479477 23196 catalog_manager.cc:4841] TS 3c348f808d02485fb3d63d4dae03cfb1 (127.20.151.1:42783): tablet ee08e8b779bd4641a869ce993c241c3d (table test-table [id=e6d09c4b07c64382a197ba4ff1bdcff1]) successfully deleted
I20260812 01:16:44.484050 24115 ts_tablet_manager.cc:1918] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:16:44.484293 24115 log.cc:1196] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: Deleting WAL directory at /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786497066589499-21084-0/minicluster-data/ts-2/wal/wals/ee08e8b779bd4641a869ce993c241c3d
I20260812 01:16:44.484869 24115 ts_tablet_manager.cc:1939] T ee08e8b779bd4641a869ce993c241c3d P 53592d1590644865beab8ccfba2074fa: Deleting consensus metadata
I20260812 01:16:44.486585 23193 catalog_manager.cc:4841] TS 53592d1590644865beab8ccfba2074fa (127.20.151.3:46815): tablet ee08e8b779bd4641a869ce993c241c3d (table test-table [id=e6d09c4b07c64382a197ba4ff1bdcff1]) successfully deleted
I20260812 01:16:44.491588 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.458585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.20.151.1:42783)
0812 01:16:44.458780 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:44.458787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:44.471169 (+ 12382us) server_negotiation.cc:407] Connection header received
0812 01:16:44.471697 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:44.471704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:44.471825 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:44.471902 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:44.472679 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.473796 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.483460 (+ 9664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.484098 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.484207 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.490048 (+ 5841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:44.490058 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:44.490073 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:44.490676 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:44.490734 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:44.490757 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:44.490866 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:44.491030 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 01:16:44.491157 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:16:44.492314 23555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:44.583900 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 23328
I20260812 01:16:44.615396 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 23566
I20260812 01:16:44.623598 24055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.616263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.20.151.2:45535)
0812 01:16:44.616461 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:44.616468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:44.616490 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:16:44.616652 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:44.616659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:44.616781 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:44.616900 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:44.617242 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.618414 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.619889 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.620572 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.620703 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.621732 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:44.621745 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:44.621764 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:44.622638 (+ 874us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:44.622706 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:44.622733 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:44.622849 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:44.623086 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 01:16:44.623192 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 01:16:44.624337 23780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:44.742874 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 23566
I20260812 01:16:44.772181 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 23797
I20260812 01:16:44.779054 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:44.773006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.20.151.3:46815)
0812 01:16:44.773192 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:44.773196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:44.773216 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:16:44.773467 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:44.773472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:44.773588 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:44.773677 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:44.774055 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.774808 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.776179 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:44.776588 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:44.776683 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:44.777667 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:44.777677 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:44.777692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:44.778279 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:44.778328 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:44.778349 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:44.778445 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:44.778654 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 01:16:44.778758 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:16:44.779748 24025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:44.869443 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 23797
I20260812 01:16:44.893175 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 23161
I20260812 01:16:44.894304 23271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:45.004465 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 23161
2026-08-12T01:16:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3158 ms)
[----------] 6 tests from EncryptionPolicyTest (24288 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:16:45 dist-test-slave-21b9 krb5kdc[24138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:16:45 dist-test-slave-21b9 krb5kdc[24138](info): set up 2 sockets
Aug 12 01:16:45 dist-test-slave-21b9 krb5kdc[24138](info): commencing operation
krb5kdc: starting...
W20260812 01:16:46.321633 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.004s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:16:46 dist-test-slave-21b9 krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:16:46.700886 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:47.058257 24163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:47.058578 24163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:47.058702 24163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:47.066195 24163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:47.066322 24163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:47.066397 24163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:47.066483 24163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:47.066538 24163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:47.077021 24163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33963
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:47.078442 24163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:47.079872 24163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:47.091857 24176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:47.091934 24172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:47.091900 24173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:47.092876 24163 server_base.cc:1034] running on GCE node
I20260812 01:16:47.093757 24163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:47.095366 24163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:47.096597 24163 hybrid_clock.cc:648] HybridClock initialized: now 1786497407096548 us; error 44 us; skew 500 ppm
I20260812 01:16:47.096971 24163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:47 dist-test-slave-21b9 krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:47.100176 24163 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:16:47.101684 24163 webserver.cc:460] Webserver started at http://127.0.0.1:46507/ using document root <none> and password file <none>
I20260812 01:16:47.102396 24163 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:47.102531 24163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:47.102840 24163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:47.106015 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "92ab7bbd2ee94953ba8da149165f1674"
format_stamp: "Formatted at 2026-08-12 01:16:47 on dist-test-slave-21b9"
server_key: "05f12fc79130a4433a886fb53d1d2cb7"
server_key_iv: "2ab9d05cad73a9a65f2dd5e5179ec5d9"
server_key_version: "encryptionkey@0"
I20260812 01:16:47.106841 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "92ab7bbd2ee94953ba8da149165f1674"
format_stamp: "Formatted at 2026-08-12 01:16:47 on dist-test-slave-21b9"
server_key: "05f12fc79130a4433a886fb53d1d2cb7"
server_key_iv: "2ab9d05cad73a9a65f2dd5e5179ec5d9"
server_key_version: "encryptionkey@0"
I20260812 01:16:47.112203 24163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 01:16:47.115796 24183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:47.117081 24163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:16:47.117267 24163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "92ab7bbd2ee94953ba8da149165f1674"
format_stamp: "Formatted at 2026-08-12 01:16:47 on dist-test-slave-21b9"
server_key: "05f12fc79130a4433a886fb53d1d2cb7"
server_key_iv: "2ab9d05cad73a9a65f2dd5e5179ec5d9"
server_key_version: "encryptionkey@0"
I20260812 01:16:47.117462 24163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:47.133047 24163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:47.137398 24163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:47.137681 24163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:47.156493 24163 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33963
I20260812 01:16:47.156517 24264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33963 every 8 connection(s)
I20260812 01:16:47.157917 24163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:47.161875 24265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:47.167843 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 24163
I20260812 01:16:47.168115 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:47.169018 21084 external_mini_cluster.cc:1428] Setting key 2fdb05edbb1a8e6910a2459f1737069d
I20260812 01:16:47.173863 24265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674: Bootstrap starting.
I20260812 01:16:47.177632 24265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:47.179179 24265 log.cc:824] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:47.182847 24265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674: No bootstrap required, opened a new log
Aug 12 01:16:47 dist-test-slave-21b9 krb5kdc[24138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 01:16:47.192147 24265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab7bbd2ee94953ba8da149165f1674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33963 } }
I20260812 01:16:47.192659 24265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:47.192795 24265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ab7bbd2ee94953ba8da149165f1674, State: Initialized, Role: FOLLOWER
I20260812 01:16:47.193519 24265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab7bbd2ee94953ba8da149165f1674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33963 } }
I20260812 01:16:47.193769 24265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:47.193920 24265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:47.194084 24265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:47.194590 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:47.171229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40697 (local address 127.0.0.1:33963)
0812 01:16:47.172138 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:47.172150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:47.172194 (+ 44us) server_negotiation.cc:407] Connection header received
0812 01:16:47.173949 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:47.173981 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:47.174406 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:47.174733 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:47.176622 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:47.178031 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:47.179997 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:47.180945 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:47.181084 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:47.186297 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:47.186307 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:47.186323 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:47.186367 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:47.189706 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:47.190816 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:47.190827 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:47.190836 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:47.191051 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:47.191565 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:47.191568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:47.191570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:47.192178 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:47.192427 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:47.193027 (+ 600us) server_negotiation.cc:299] Negotiation successful
0812 01:16:47.193383 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":420,"threads_started":1}
I20260812 01:16:47.196707 24265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab7bbd2ee94953ba8da149165f1674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33963 } }
I20260812 01:16:47.197247 24265 leader_election.cc:302] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92ab7bbd2ee94953ba8da149165f1674; no voters:
I20260812 01:16:47.197845 24265 leader_election.cc:288] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:47.197980 24273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:47.198899 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:16:47.199875 24265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:47.205215 24273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [term 1 LEADER]: Becoming Leader. State: Replica: 92ab7bbd2ee94953ba8da149165f1674, State: Running, Role: LEADER
I20260812 01:16:47.205955 24273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab7bbd2ee94953ba8da149165f1674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33963 } }
I20260812 01:16:47.208910 24274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92ab7bbd2ee94953ba8da149165f1674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab7bbd2ee94953ba8da149165f1674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33963 } } }
I20260812 01:16:47.209329 24274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:47.210611 24275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92ab7bbd2ee94953ba8da149165f1674. Latest consensus state: current_term: 1 leader_uuid: "92ab7bbd2ee94953ba8da149165f1674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ab7bbd2ee94953ba8da149165f1674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33963 } } }
I20260812 01:16:47.210815 24301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:47.210906 24275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:47.216063 24301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:47.224511 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:47.206908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39575 (local address 127.0.0.1:33963)
0812 01:16:47.207160 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:47.207168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:47.207651 (+ 483us) server_negotiation.cc:407] Connection header received
0812 01:16:47.207855 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:47.207860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:47.207939 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:47.208046 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:47.210072 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:47.211371 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:47.213355 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:47.213994 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:47.214148 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:47.217343 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:47.217350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:47.217356 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:47.217397 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:47.220915 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:47.221791 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:47.221796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:47.221798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:47.221921 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:47.222429 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:47.222434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:47.222438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:47.222874 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:47.223155 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:47.223625 (+ 470us) server_negotiation.cc:299] Negotiation successful
0812 01:16:47.223832 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 01:16:47.229722 24301 catalog_manager.cc:1347] Generated new cluster ID: 3ad869819ee74615b38fd2a4ca48ca5f
I20260812 01:16:47.229862 24301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:47.245800 24301 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:47.247726 24301 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:47.260120 24301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92ab7bbd2ee94953ba8da149165f1674: Generated new TSK 0
I20260812 01:16:47.260995 24301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:16:47.278582 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24163
I20260812 01:16:47.279850 24260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:47.373570 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 24163
W20260812 01:16:47.397265 24132 connection.cc:535] client connection to 127.0.0.1:33963 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33963): Connection reset by peer (error 104)
2026-08-12T01:16:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:16:47 dist-test-slave-21b9 krb5kdc[24317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:16:47 dist-test-slave-21b9 krb5kdc[24317](info): set up 2 sockets
Aug 12 01:16:47 dist-test-slave-21b9 krb5kdc[24317](info): commencing operation
krb5kdc: starting...
W20260812 01:16:48.668325 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:16:48 dist-test-slave-21b9 krb5kdc[24317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:16:49.047057 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:49.399786 24342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:49.400095 24342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:49.400219 24342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:49.407564 24342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:49.407685 24342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:49.407755 24342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:49.407845 24342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:49.407909 24342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:49.418329 24342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:49.420022 24342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:49.421348 24342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:49.432919 24354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:49.432964 24355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:49.433060 24357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:49.434188 24342 server_base.cc:1034] running on GCE node
I20260812 01:16:49.435006 24342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:49.436486 24342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:49.437717 24342 hybrid_clock.cc:648] HybridClock initialized: now 1786497409437653 us; error 43 us; skew 500 ppm
I20260812 01:16:49.438082 24342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:49 dist-test-slave-21b9 krb5kdc[24317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:49.441257 24342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:16:49.442725 24342 webserver.cc:460] Webserver started at http://127.0.0.1:43231/ using document root <none> and password file <none>
I20260812 01:16:49.443493 24342 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:49.443620 24342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:49.443914 24342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:49.447139 24342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "de785697e70b4cd19e3c3e16019782e3"
format_stamp: "Formatted at 2026-08-12 01:16:49 on dist-test-slave-21b9"
server_key: "61b2f5b118db43d639b3453763587c3f"
server_key_iv: "61c2a72a466074ed577fa69969a85623"
server_key_version: "encryptionkey@0"
I20260812 01:16:49.447956 24342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "de785697e70b4cd19e3c3e16019782e3"
format_stamp: "Formatted at 2026-08-12 01:16:49 on dist-test-slave-21b9"
server_key: "61b2f5b118db43d639b3453763587c3f"
server_key_iv: "61c2a72a466074ed577fa69969a85623"
server_key_version: "encryptionkey@0"
I20260812 01:16:49.453330 24342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:16:49.456938 24369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:49.458317 24342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 01:16:49.458504 24342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "de785697e70b4cd19e3c3e16019782e3"
format_stamp: "Formatted at 2026-08-12 01:16:49 on dist-test-slave-21b9"
server_key: "61b2f5b118db43d639b3453763587c3f"
server_key_iv: "61c2a72a466074ed577fa69969a85623"
server_key_version: "encryptionkey@0"
I20260812 01:16:49.458701 24342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:49.482844 24342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:49.487442 24342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:49.487706 24342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:49.506188 24342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41867
I20260812 01:16:49.506219 24455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41867 every 8 connection(s)
I20260812 01:16:49.507710 24342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:49.511682 24456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:49.514559 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 24342
I20260812 01:16:49.514715 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:49.515162 21084 external_mini_cluster.cc:1428] Setting key 4b98df9b32f169fc13996f1d49725615
I20260812 01:16:49.524966 24456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3: Bootstrap starting.
I20260812 01:16:49.528575 24456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:49.529629 24456 log.cc:824] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:49.532469 24456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3: No bootstrap required, opened a new log
I20260812 01:16:49.535823 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:49.517054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56771 (local address 127.0.0.1:41867)
0812 01:16:49.517913 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:49.517929 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:49.517982 (+ 53us) server_negotiation.cc:407] Connection header received
0812 01:16:49.520345 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:49.520382 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:49.520928 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:49.521375 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:49.521902 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:49.523999 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:49.525410 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:49.526411 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:49.526611 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:49.532864 (+ 6253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:49.532877 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:49.532902 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:49.533637 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:49.533737 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:49.533781 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:49.533968 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:49.534239 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 01:16:49.534426 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":346,"threads_started":1}
I20260812 01:16:49.539765 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:16:49.541954 24456 raft_consensus.cc:348] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de785697e70b4cd19e3c3e16019782e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41867 } }
I20260812 01:16:49.542431 24456 raft_consensus.cc:374] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:49.542543 24456 raft_consensus.cc:729] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de785697e70b4cd19e3c3e16019782e3, State: Initialized, Role: FOLLOWER
I20260812 01:16:49.543220 24456 consensus_queue.cc:260] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de785697e70b4cd19e3c3e16019782e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41867 } }
I20260812 01:16:49.543582 24456 raft_consensus.cc:388] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:49.543735 24456 raft_consensus.cc:482] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:49.543937 24456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:49.546964 24456 raft_consensus.cc:504] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de785697e70b4cd19e3c3e16019782e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41867 } }
I20260812 01:16:49.547588 24456 leader_election.cc:302] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de785697e70b4cd19e3c3e16019782e3; no voters:
I20260812 01:16:49.548243 24456 leader_election.cc:288] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:49.548355 24475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:49.550722 24456 sys_catalog.cc:564] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:49.551568 24475 raft_consensus.cc:686] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [term 1 LEADER]: Becoming Leader. State: Replica: de785697e70b4cd19e3c3e16019782e3, State: Running, Role: LEADER
I20260812 01:16:49.551942 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:49.545146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57675 (local address 127.0.0.1:41867)
0812 01:16:49.545327 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:49.545334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:49.545663 (+ 329us) server_negotiation.cc:407] Connection header received
0812 01:16:49.545805 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:49.545810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:49.545889 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:49.546005 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:49.546437 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:49.547511 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:49.549080 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:49.549690 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:49.549768 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:49.551043 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:49.551049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:49.551052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:49.551072 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:49.551110 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:49.551126 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:49.551218 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:49.551536 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 01:16:49.551608 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 01:16:49.552232 24475 consensus_queue.cc:237] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de785697e70b4cd19e3c3e16019782e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41867 } }
I20260812 01:16:49.555330 24477 sys_catalog.cc:455] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de785697e70b4cd19e3c3e16019782e3. Latest consensus state: current_term: 1 leader_uuid: "de785697e70b4cd19e3c3e16019782e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de785697e70b4cd19e3c3e16019782e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41867 } } }
I20260812 01:16:49.555433 24476 sys_catalog.cc:455] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de785697e70b4cd19e3c3e16019782e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de785697e70b4cd19e3c3e16019782e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41867 } } }
I20260812 01:16:49.555821 24476 sys_catalog.cc:458] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:49.557566 24477 sys_catalog.cc:458] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:49.557914 24483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:49.564038 24483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:49.574112 24483 catalog_manager.cc:1347] Generated new cluster ID: cbab9d41c5684353989c3c92d4ac9585
I20260812 01:16:49.574241 24483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:49.594527 24483 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:49.595947 24483 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:49.607537 24483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de785697e70b4cd19e3c3e16019782e3: Generated new TSK 0
I20260812 01:16:49.608453 24483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:16:49.615851 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24342
W20260812 01:16:49.617609 24445 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56771
W20260812 01:16:49.618463 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24342: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:16:49.618641 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 24342
W20260812 01:16:49.645992 24312 connection.cc:535] client connection to 127.0.0.1:41867 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41867): Connection reset by peer (error 104)
2026-08-12T01:16:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:16:49 dist-test-slave-21b9 krb5kdc[24516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:16:49 dist-test-slave-21b9 krb5kdc[24516](info): set up 2 sockets
Aug 12 01:16:49 dist-test-slave-21b9 krb5kdc[24516](info): commencing operation
krb5kdc: starting...
W20260812 01:16:50.924815 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.000s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:16:51 dist-test-slave-21b9 krb5kdc[24516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:16:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:16:51.301153 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:51.657624 24549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:51.657945 24549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:51.658092 24549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:51.665395 24549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:51.665547 24549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:51.665642 24549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:51.665727 24549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:51.665786 24549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:51.675830 24549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:51.677172 24549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:51.678510 24549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:51.690165 24556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:51.690670 24558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:51.691112 24561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:51.691428 24549 server_base.cc:1034] running on GCE node
I20260812 01:16:51.692400 24549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:51.693822 24549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:51.695036 24549 hybrid_clock.cc:648] HybridClock initialized: now 1786497411694980 us; error 53 us; skew 500 ppm
I20260812 01:16:51.695420 24549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:51 dist-test-slave-21b9 krb5kdc[24516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:51.698773 24549 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:16:51.700361 24549 webserver.cc:460] Webserver started at http://127.0.0.1:35997/ using document root <none> and password file <none>
I20260812 01:16:51.701067 24549 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:51.701191 24549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:51.701488 24549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:51.704599 24549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "2250532811284815b26b271ff4d389de"
format_stamp: "Formatted at 2026-08-12 01:16:51 on dist-test-slave-21b9"
server_key: "22b19ea60f79e4b310d205c71121e349"
server_key_iv: "3c41d4d26d5edea8bc5992304035f86a"
server_key_version: "encryptionkey@0"
I20260812 01:16:51.705379 24549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "2250532811284815b26b271ff4d389de"
format_stamp: "Formatted at 2026-08-12 01:16:51 on dist-test-slave-21b9"
server_key: "22b19ea60f79e4b310d205c71121e349"
server_key_iv: "3c41d4d26d5edea8bc5992304035f86a"
server_key_version: "encryptionkey@0"
I20260812 01:16:51.710829 24549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 01:16:51.714478 24573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:51.715934 24549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:16:51.716161 24549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "2250532811284815b26b271ff4d389de"
format_stamp: "Formatted at 2026-08-12 01:16:51 on dist-test-slave-21b9"
server_key: "22b19ea60f79e4b310d205c71121e349"
server_key_iv: "3c41d4d26d5edea8bc5992304035f86a"
server_key_version: "encryptionkey@0"
I20260812 01:16:51.716415 24549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:51.722512 24549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:51.727041 24549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:51.727375 24549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:51.745946 24549 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35267
I20260812 01:16:51.746002 24659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35267 every 8 connection(s)
I20260812 01:16:51.747489 24549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:51.747910 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 24549
I20260812 01:16:51.748066 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:51.748602 21084 external_mini_cluster.cc:1428] Setting key 089bb48c2553ce993af82fed3b0bc963
I20260812 01:16:51.752218 24661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:51.756521 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:51.750541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35721 (local address 127.0.0.1:35267)
0812 01:16:51.751372 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:51.751385 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:51.751428 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:16:51.753164 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:51.753187 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:51.753509 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:51.753865 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:51.754893 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:51.754900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:51.754917 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:51.754932 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:51.755015 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:51.755035 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:51.755292 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:51.755426 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 01:16:51.755608 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":347,"threads_started":1}
I20260812 01:16:51.760128 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:16:51.763515 24661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de: Bootstrap starting.
I20260812 01:16:51.767000 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:51.764430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51431 (local address 127.0.0.1:35267)
0812 01:16:51.764570 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:51.764575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:51.765125 (+ 550us) server_negotiation.cc:407] Connection header received
0812 01:16:51.765303 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:51.765310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:51.765381 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:51.765503 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:51.766356 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:51.766361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:51.766365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:51.766384 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:51.766431 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:51.766462 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:51.766577 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:51.766662 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 01:16:51.766718 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 01:16:51.767321 24661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:51.768455 24661 log.cc:824] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:51.771093 24661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de: No bootstrap required, opened a new log
I20260812 01:16:51.778472 24661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2250532811284815b26b271ff4d389de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260812 01:16:51.778892 24661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:51.779021 24661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2250532811284815b26b271ff4d389de, State: Initialized, Role: FOLLOWER
I20260812 01:16:51.779686 24661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2250532811284815b26b271ff4d389de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260812 01:16:51.779906 24661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:51.780076 24661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:51.780236 24661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:51.782439 24661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2250532811284815b26b271ff4d389de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260812 01:16:51.782960 24661 leader_election.cc:302] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2250532811284815b26b271ff4d389de; no voters:
I20260812 01:16:51.783610 24661 leader_election.cc:288] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:51.783739 24681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:51.784390 24681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [term 1 LEADER]: Becoming Leader. State: Replica: 2250532811284815b26b271ff4d389de, State: Running, Role: LEADER
I20260812 01:16:51.784976 24681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2250532811284815b26b271ff4d389de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260812 01:16:51.785701 24661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:51.787801 24683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2250532811284815b26b271ff4d389de. Latest consensus state: current_term: 1 leader_uuid: "2250532811284815b26b271ff4d389de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2250532811284815b26b271ff4d389de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } } }
I20260812 01:16:51.788201 24683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:51.788973 24682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2250532811284815b26b271ff4d389de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2250532811284815b26b271ff4d389de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } } }
I20260812 01:16:51.789204 24690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:51.789326 24682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:51.795943 24690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:51.805500 24690 catalog_manager.cc:1347] Generated new cluster ID: 06e8671e8cfc438e9353a611d2ab7bfc
I20260812 01:16:51.805634 24690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:51.839187 24690 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:51.839761 24690 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:51.860109 24690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2250532811284815b26b271ff4d389de: Generated new TSK 0
I20260812 01:16:51.861227 24690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:16:51.866431 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24549
W20260812 01:16:51.867591 24652 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35721
W20260812 01:16:51.868270 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24549: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:16:51.868427 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 24549
2026-08-12T01:16:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:16:52 dist-test-slave-21b9 krb5kdc[24718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:16:52 dist-test-slave-21b9 krb5kdc[24718](info): set up 2 sockets
Aug 12 01:16:52 dist-test-slave-21b9 krb5kdc[24718](info): commencing operation
krb5kdc: starting...
W20260812 01:16:53.187055 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.000s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:16:53 dist-test-slave-21b9 krb5kdc[24718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:16:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:16:53.558904 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:53.915668 24737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:53.915980 24737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:53.916131 24737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:53.923475 24737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:53.923606 24737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:53.923664 24737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:53.923749 24737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:53.923815 24737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:53.933899 24737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:53.935222 24737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:53.936609 24737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:53.948657 24747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:53.948658 24752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:53.948632 24749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:53.949483 24737 server_base.cc:1034] running on GCE node
I20260812 01:16:53.950325 24737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:53.951874 24737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:53.953090 24737 hybrid_clock.cc:648] HybridClock initialized: now 1786497413953036 us; error 42 us; skew 500 ppm
I20260812 01:16:53.953492 24737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:53 dist-test-slave-21b9 krb5kdc[24718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:53.956738 24737 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:16:53.958236 24737 webserver.cc:460] Webserver started at http://127.0.0.1:41409/ using document root <none> and password file <none>
I20260812 01:16:53.958918 24737 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:53.959044 24737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:53.959403 24737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:53.962453 24737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "3ada3e263a6e42f2a07cecd45134dc60"
format_stamp: "Formatted at 2026-08-12 01:16:53 on dist-test-slave-21b9"
server_key: "b24fe2479399e0297604773951dc4dcb"
server_key_iv: "3dd11678039d8892945bb2163670f9d1"
server_key_version: "encryptionkey@0"
I20260812 01:16:53.963254 24737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "3ada3e263a6e42f2a07cecd45134dc60"
format_stamp: "Formatted at 2026-08-12 01:16:53 on dist-test-slave-21b9"
server_key: "b24fe2479399e0297604773951dc4dcb"
server_key_iv: "3dd11678039d8892945bb2163670f9d1"
server_key_version: "encryptionkey@0"
I20260812 01:16:53.968894 24737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260812 01:16:53.973608 24763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:53.975112 24737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 01:16:53.975335 24737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "3ada3e263a6e42f2a07cecd45134dc60"
format_stamp: "Formatted at 2026-08-12 01:16:53 on dist-test-slave-21b9"
server_key: "b24fe2479399e0297604773951dc4dcb"
server_key_iv: "3dd11678039d8892945bb2163670f9d1"
server_key_version: "encryptionkey@0"
I20260812 01:16:53.975564 24737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:54.003551 24737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:54.007949 24737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:54.008248 24737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:54.032483 24737 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40281
I20260812 01:16:54.032478 24845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40281 every 8 connection(s)
I20260812 01:16:54.033953 24737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:54.036458 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 24737
I20260812 01:16:54.036667 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:54.037243 21084 external_mini_cluster.cc:1428] Setting key 9865c86db9b3ca035c2e5d137bf667e1
I20260812 01:16:54.038610 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 01:16:54 dist-test-slave-21b9 krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 01:16:54.052830 24846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60: Bootstrap starting.
I20260812 01:16:54.056695 24846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:54.058037 24846 log.cc:824] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:54.061591 24846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60: No bootstrap required, opened a new log
I20260812 01:16:54.062074 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:54.039125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45741 (local address 127.0.0.1:40281)
0812 01:16:54.039962 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:54.039977 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:54.040030 (+ 53us) server_negotiation.cc:407] Connection header received
0812 01:16:54.041985 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:54.042015 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:54.042562 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:54.043045 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:54.044538 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:54.046607 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:54.047978 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:54.049027 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:54.049240 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:54.053823 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:54.053832 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:54.053850 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:54.053895 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:54.057101 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:54.058132 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:54.058141 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:54.058150 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:54.058380 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:54.058921 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:54.058927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:54.058930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:54.059717 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:54.059963 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:54.060552 (+ 589us) server_negotiation.cc:299] Negotiation successful
0812 01:16:54.060975 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":343,"threads_started":1}
I20260812 01:16:54.065927 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:16:54.071552 24846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada3e263a6e42f2a07cecd45134dc60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260812 01:16:54.072078 24846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:54.072175 24846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ada3e263a6e42f2a07cecd45134dc60, State: Initialized, Role: FOLLOWER
I20260812 01:16:54.072814 24846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada3e263a6e42f2a07cecd45134dc60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260812 01:16:54.073057 24846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:54.073192 24846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:54.073356 24846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:54.075768 24846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada3e263a6e42f2a07cecd45134dc60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260812 01:16:54.076318 24846 leader_election.cc:302] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ada3e263a6e42f2a07cecd45134dc60; no voters:
I20260812 01:16:54.076949 24846 leader_election.cc:288] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:54.077111 24862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:54.078233 24862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [term 1 LEADER]: Becoming Leader. State: Replica: 3ada3e263a6e42f2a07cecd45134dc60, State: Running, Role: LEADER
I20260812 01:16:54.078864 24862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada3e263a6e42f2a07cecd45134dc60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260812 01:16:54.079600 24846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:54.081609 24869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ada3e263a6e42f2a07cecd45134dc60. Latest consensus state: current_term: 1 leader_uuid: "3ada3e263a6e42f2a07cecd45134dc60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada3e263a6e42f2a07cecd45134dc60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } } }
I20260812 01:16:54.082042 24869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:54.082633 24866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ada3e263a6e42f2a07cecd45134dc60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada3e263a6e42f2a07cecd45134dc60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } } }
I20260812 01:16:54.082963 24866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:54.085335 24874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:54.089777 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:54.071692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32941 (local address 127.0.0.1:40281)
0812 01:16:54.071898 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:54.071913 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:54.072248 (+ 335us) server_negotiation.cc:407] Connection header received
0812 01:16:54.072477 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:54.072481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:54.072552 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:54.072638 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:54.074306 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:54.075063 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:54.076434 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:54.077069 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:54.077207 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:54.080566 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:54.080572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:54.080577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:16:54.080629 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:54.085052 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:54.086155 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:54.086162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:54.086166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:54.086355 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:16:54.086857 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:16:54.086862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:16:54.086866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:16:54.087355 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:54.087654 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:54.089048 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0812 01:16:54.089349 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 01:16:54.094899 24874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:54.104807 24874 catalog_manager.cc:1347] Generated new cluster ID: 534554d58c404823b6b24d6e537939dd
I20260812 01:16:54.104949 24874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:54.135847 24874 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:54.137269 24874 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:54.145532 24874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ada3e263a6e42f2a07cecd45134dc60: Generated new TSK 0
I20260812 01:16:54.146363 24874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:16:54.161393 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24737
I20260812 01:16:54.162894 24839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:16:54.258260 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 24737
2026-08-12T01:16:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:16:54 dist-test-slave-21b9 krb5kdc[24904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:16:54 dist-test-slave-21b9 krb5kdc[24904](info): set up 2 sockets
Aug 12 01:16:54 dist-test-slave-21b9 krb5kdc[24904](info): commencing operation
krb5kdc: starting...
W20260812 01:16:55.562690 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.003s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:16:55 dist-test-slave-21b9 krb5kdc[24904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:16:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:16:55.938446 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:56.298712 24929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:56.299130 24929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:56.299307 24929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:56.310288 24929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:56.310457 24929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:56.310540 24929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:56.310631 24929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:56.310693 24929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:56.324451 24929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:56.326210 24929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:56.328027 24929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:56.341717 24935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:56.341691 24936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:56.342377 24938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:56.343003 24929 server_base.cc:1034] running on GCE node
I20260812 01:16:56.344117 24929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:56.346026 24929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:56.347230 24929 hybrid_clock.cc:648] HybridClock initialized: now 1786497416347184 us; error 44 us; skew 500 ppm
I20260812 01:16:56.347671 24929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:56 dist-test-slave-21b9 krb5kdc[24904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:56.351037 24929 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:16:56.352586 24929 webserver.cc:460] Webserver started at http://127.0.0.1:41309/ using document root <none> and password file <none>
I20260812 01:16:56.353256 24929 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:56.353367 24929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:56.353716 24929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:56.358016 24929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "226d0e3b981842a3b0e36ffb720504a8"
format_stamp: "Formatted at 2026-08-12 01:16:56 on dist-test-slave-21b9"
server_key: "9e3cdebcba2c4dee24765a893992482d"
server_key_iv: "3a61c525ba1f6082c3f0cc49fc75532c"
server_key_version: "encryptionkey@0"
I20260812 01:16:56.358973 24929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "226d0e3b981842a3b0e36ffb720504a8"
format_stamp: "Formatted at 2026-08-12 01:16:56 on dist-test-slave-21b9"
server_key: "9e3cdebcba2c4dee24765a893992482d"
server_key_iv: "3a61c525ba1f6082c3f0cc49fc75532c"
server_key_version: "encryptionkey@0"
I20260812 01:16:56.364609 24929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:16:56.368264 24949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:56.369768 24929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:16:56.369935 24929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "226d0e3b981842a3b0e36ffb720504a8"
format_stamp: "Formatted at 2026-08-12 01:16:56 on dist-test-slave-21b9"
server_key: "9e3cdebcba2c4dee24765a893992482d"
server_key_iv: "3a61c525ba1f6082c3f0cc49fc75532c"
server_key_version: "encryptionkey@0"
I20260812 01:16:56.370121 24929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:56.388656 24929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:56.393127 24929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:56.393378 24929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:56.411701 24929 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38689
I20260812 01:16:56.411751 25045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38689 every 8 connection(s)
I20260812 01:16:56.413211 24929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:56.417212 25046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:56.420058 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 24929
I20260812 01:16:56.420250 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:56.420809 21084 external_mini_cluster.cc:1428] Setting key b416f496900667c40e5c70a313b86207
I20260812 01:16:56.429409 25046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8: Bootstrap starting.
I20260812 01:16:56.433475 25046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:56.434773 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:56.422838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35919 (local address 127.0.0.1:38689)
0812 01:16:56.423695 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:56.423707 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:56.423752 (+ 45us) server_negotiation.cc:407] Connection header received
0812 01:16:56.425459 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:56.425483 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:56.425878 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:56.426264 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:56.426835 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:56.428249 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:56.429592 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:56.430621 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:56.430827 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:56.431944 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:56.431955 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:56.431973 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:56.432716 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:56.432818 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:56.432857 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:56.432991 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:56.433118 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 01:16:56.433401 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":386,"threads_started":1}
I20260812 01:16:56.434871 25046 log.cc:824] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:56.438120 25046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8: No bootstrap required, opened a new log
I20260812 01:16:56.438972 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:16:56.448346 25046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d0e3b981842a3b0e36ffb720504a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38689 } }
I20260812 01:16:56.448911 25046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:56.449055 25046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226d0e3b981842a3b0e36ffb720504a8, State: Initialized, Role: FOLLOWER
I20260812 01:16:56.449708 25046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d0e3b981842a3b0e36ffb720504a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38689 } }
I20260812 01:16:56.449950 25046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:56.450115 25046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:56.450321 25046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:56.450837 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:56.444405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59595 (local address 127.0.0.1:38689)
0812 01:16:56.444601 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:56.444611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:56.444935 (+ 324us) server_negotiation.cc:407] Connection header received
0812 01:16:56.445088 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:56.445093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:56.445182 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:56.445282 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:56.445788 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:56.446899 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:56.448235 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:16:56.448866 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:16:56.449004 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:16:56.450054 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:56.450060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:56.450065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:56.450089 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:56.450142 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:56.450164 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:56.450255 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:56.450426 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 01:16:56.450523 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:16:56.453222 25046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d0e3b981842a3b0e36ffb720504a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38689 } }
I20260812 01:16:56.453724 25046 leader_election.cc:302] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 226d0e3b981842a3b0e36ffb720504a8; no voters:
I20260812 01:16:56.454273 25046 leader_election.cc:288] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:56.454429 25061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:56.455170 25061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [term 1 LEADER]: Becoming Leader. State: Replica: 226d0e3b981842a3b0e36ffb720504a8, State: Running, Role: LEADER
I20260812 01:16:56.455768 25061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d0e3b981842a3b0e36ffb720504a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38689 } }
I20260812 01:16:56.456468 25046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:56.459054 25062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "226d0e3b981842a3b0e36ffb720504a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d0e3b981842a3b0e36ffb720504a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38689 } } }
I20260812 01:16:56.459650 25062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:56.459081 25065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 226d0e3b981842a3b0e36ffb720504a8. Latest consensus state: current_term: 1 leader_uuid: "226d0e3b981842a3b0e36ffb720504a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d0e3b981842a3b0e36ffb720504a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38689 } } }
I20260812 01:16:56.460108 25065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8 [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:56.467816 25073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:56.472808 25073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:56.481838 25073 catalog_manager.cc:1347] Generated new cluster ID: 75275caaf0c9489a8807bfed74f18173
I20260812 01:16:56.481972 25073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:56.491462 25073 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:56.493472 25073 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:56.509903 25073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 226d0e3b981842a3b0e36ffb720504a8: Generated new TSK 0
I20260812 01:16:56.511166 25073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:16:56.524243 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24929
W20260812 01:16:56.525478 25039 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35919
W20260812 01:16:56.526291 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 24929: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:16:56.526463 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 24929
2026-08-12T01:16:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:16:56 dist-test-slave-21b9 krb5kdc[25103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:16:56 dist-test-slave-21b9 krb5kdc[25103](info): set up 2 sockets
Aug 12 01:16:56 dist-test-slave-21b9 krb5kdc[25103](info): commencing operation
krb5kdc: starting...
W20260812 01:16:57.826023 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.005s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:16:58 dist-test-slave-21b9 krb5kdc[25103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:16:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:16:58.203788 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:16:58.554361 25118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:16:58.554683 25118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:16:58.554817 25118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:16:58.562181 25118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:16:58.562299 25118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:16:58.562371 25118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:16:58.562455 25118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:16:58.562495 25118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:16:58.572849 25118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:16:58.574193 25118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:16:58.575562 25118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:16:58.587189 25125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:58.587232 25126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:16:58.587145 25128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:16:58.588482 25118 server_base.cc:1034] running on GCE node
I20260812 01:16:58.589313 25118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:16:58.590745 25118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:16:58.591979 25118 hybrid_clock.cc:648] HybridClock initialized: now 1786497418591921 us; error 38 us; skew 500 ppm
I20260812 01:16:58.592345 25118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:16:58 dist-test-slave-21b9 krb5kdc[25103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:16:58.595815 25118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:16:58.597353 25118 webserver.cc:460] Webserver started at http://127.0.0.1:34789/ using document root <none> and password file <none>
I20260812 01:16:58.598032 25118 fs_manager.cc:359] Metadata directory not provided
I20260812 01:16:58.598156 25118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:16:58.598464 25118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:16:58.601509 25118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "4643d6814a6a4aeea857cc51c36ab5fa"
format_stamp: "Formatted at 2026-08-12 01:16:58 on dist-test-slave-21b9"
server_key: "85b274dc1605df8dfda1a910aa15317b"
server_key_iv: "4d881a95b3c4be039813ccb6c2d692cc"
server_key_version: "encryptionkey@0"
I20260812 01:16:58.602250 25118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "4643d6814a6a4aeea857cc51c36ab5fa"
format_stamp: "Formatted at 2026-08-12 01:16:58 on dist-test-slave-21b9"
server_key: "85b274dc1605df8dfda1a910aa15317b"
server_key_iv: "4d881a95b3c4be039813ccb6c2d692cc"
server_key_version: "encryptionkey@0"
I20260812 01:16:58.607372 25118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260812 01:16:58.610872 25141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:16:58.612174 25118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 01:16:58.612337 25118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "4643d6814a6a4aeea857cc51c36ab5fa"
format_stamp: "Formatted at 2026-08-12 01:16:58 on dist-test-slave-21b9"
server_key: "85b274dc1605df8dfda1a910aa15317b"
server_key_iv: "4d881a95b3c4be039813ccb6c2d692cc"
server_key_version: "encryptionkey@0"
I20260812 01:16:58.612527 25118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:16:58.618312 25118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:16:58.622248 25118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:16:58.622526 25118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:16:58.640892 25118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45887
I20260812 01:16:58.640933 25234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45887 every 8 connection(s)
I20260812 01:16:58.642333 25118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:16:58.646368 25235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:16:58.650522 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 25118
I20260812 01:16:58.650722 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:16:58.651309 21084 external_mini_cluster.cc:1428] Setting key af985ef63c2ff5a7d78b833a803f1b51
I20260812 01:16:58.659356 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:58.653220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59509 (local address 127.0.0.1:45887)
0812 01:16:58.654077 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:58.654089 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:58.654132 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:16:58.655845 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:58.655872 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:58.656257 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:58.656606 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:58.657772 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:58.657778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:58.657792 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:58.657807 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:58.657872 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:58.657892 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:58.658076 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:58.658175 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 01:16:58.658359 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":388,"threads_started":1}
I20260812 01:16:58.660238 25235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa: Bootstrap starting.
I20260812 01:16:58.663302 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:16:58.664052 25235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa: Neither blocks nor log segments found. Creating new log.
I20260812 01:16:58.665175 25235 log.cc:824] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa: Log is configured to *not* fsync() on all Append() calls
I20260812 01:16:58.668921 25235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa: No bootstrap required, opened a new log
I20260812 01:16:58.670449 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:16:58.668034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36663 (local address 127.0.0.1:45887)
0812 01:16:58.668214 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 01:16:58.668221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:16:58.668592 (+ 371us) server_negotiation.cc:407] Connection header received
0812 01:16:58.668764 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:16:58.668769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:16:58.668818 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:16:58.668903 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:16:58.669829 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:16:58.669834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:16:58.669837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:16:58.669852 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:16:58.669892 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:16:58.669907 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:16:58.670003 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 01:16:58.670130 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 01:16:58.670193 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 01:16:58.677115 25235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643d6814a6a4aeea857cc51c36ab5fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45887 } }
I20260812 01:16:58.677515 25235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:16:58.677613 25235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4643d6814a6a4aeea857cc51c36ab5fa, State: Initialized, Role: FOLLOWER
I20260812 01:16:58.678189 25235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643d6814a6a4aeea857cc51c36ab5fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45887 } }
I20260812 01:16:58.678454 25235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:16:58.678589 25235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:16:58.678742 25235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:16:58.681342 25235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643d6814a6a4aeea857cc51c36ab5fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45887 } }
I20260812 01:16:58.681840 25235 leader_election.cc:302] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4643d6814a6a4aeea857cc51c36ab5fa; no voters:
I20260812 01:16:58.682422 25235 leader_election.cc:288] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:16:58.682535 25253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:16:58.683184 25253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [term 1 LEADER]: Becoming Leader. State: Replica: 4643d6814a6a4aeea857cc51c36ab5fa, State: Running, Role: LEADER
I20260812 01:16:58.683826 25253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643d6814a6a4aeea857cc51c36ab5fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45887 } }
I20260812 01:16:58.684691 25235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:16:58.686746 25255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4643d6814a6a4aeea857cc51c36ab5fa. Latest consensus state: current_term: 1 leader_uuid: "4643d6814a6a4aeea857cc51c36ab5fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643d6814a6a4aeea857cc51c36ab5fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45887 } } }
I20260812 01:16:58.687143 25255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:58.686848 25254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4643d6814a6a4aeea857cc51c36ab5fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4643d6814a6a4aeea857cc51c36ab5fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45887 } } }
I20260812 01:16:58.687486 25254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa [sys.catalog]: This master's current role is: LEADER
I20260812 01:16:58.689237 25263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:16:58.694690 25263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:16:58.705003 25263 catalog_manager.cc:1347] Generated new cluster ID: 533c49ceeb93482f9cc2105bcd9689dc
I20260812 01:16:58.705144 25263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:16:58.727770 25263 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:16:58.728245 25263 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:16:58.737615 25263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4643d6814a6a4aeea857cc51c36ab5fa: Generated new TSK 0
I20260812 01:16:58.738495 25263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:16:58.754575 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25118
W20260812 01:16:58.755894 25226 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59509
W20260812 01:16:58.756616 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25118: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:16:58.756781 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 25118
2026-08-12T01:16:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:16:58 dist-test-slave-21b9 krb5kdc[25288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:16:58 dist-test-slave-21b9 krb5kdc[25288](info): set up 2 sockets
Aug 12 01:16:58 dist-test-slave-21b9 krb5kdc[25288](info): commencing operation
krb5kdc: starting...
W20260812 01:17:00.062245 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.002s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:00 dist-test-slave-21b9 krb5kdc[25288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:17:00.442638 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:00.800992 25311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:00.801302 25311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:00.801399 25311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:00.808785 25311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:00.808925 25311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:00.808982 25311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:00.809046 25311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:00.809090 25311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:00.819140 25311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:00.820495 25311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:00.821806 25311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:00.833586 25320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:00.833549 25321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:00.833675 25323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:00.834614 25311 server_base.cc:1034] running on GCE node
I20260812 01:17:00.835471 25311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:00.836916 25311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:00.838137 25311 hybrid_clock.cc:648] HybridClock initialized: now 1786497420838094 us; error 32 us; skew 500 ppm
I20260812 01:17:00.838490 25311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:00 dist-test-slave-21b9 krb5kdc[25288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:00.841594 25311 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:17:00.843134 25311 webserver.cc:460] Webserver started at http://127.0.0.1:34913/ using document root <none> and password file <none>
I20260812 01:17:00.843864 25311 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:00.843973 25311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:00.844233 25311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:00.847306 25311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "4d7600b7d9704961bd44afd87a045e9c"
format_stamp: "Formatted at 2026-08-12 01:17:00 on dist-test-slave-21b9"
server_key: "6440ee372acaa550a6db93c3c024b441"
server_key_iv: "adf160fac04d1656f3494a274ce73452"
server_key_version: "encryptionkey@0"
I20260812 01:17:00.848163 25311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "4d7600b7d9704961bd44afd87a045e9c"
format_stamp: "Formatted at 2026-08-12 01:17:00 on dist-test-slave-21b9"
server_key: "6440ee372acaa550a6db93c3c024b441"
server_key_iv: "adf160fac04d1656f3494a274ce73452"
server_key_version: "encryptionkey@0"
I20260812 01:17:00.853518 25311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:17:00.857223 25337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:00.858639 25311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:17:00.858819 25311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "4d7600b7d9704961bd44afd87a045e9c"
format_stamp: "Formatted at 2026-08-12 01:17:00 on dist-test-slave-21b9"
server_key: "6440ee372acaa550a6db93c3c024b441"
server_key_iv: "adf160fac04d1656f3494a274ce73452"
server_key_version: "encryptionkey@0"
I20260812 01:17:00.859018 25311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:00.883095 25311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:00.887578 25311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:00.887848 25311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:00.906965 25311 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41705
I20260812 01:17:00.906998 25435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41705 every 8 connection(s)
I20260812 01:17:00.908427 25311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:00.910674 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 25311
I20260812 01:17:00.910856 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:00.911335 21084 external_mini_cluster.cc:1428] Setting key 4e6ac41d00e08f7a8cf1b9e9ea0e9e6b
I20260812 01:17:00.912726 25436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 01:17:00 dist-test-slave-21b9 krb5kdc[25288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 01:17:00.926959 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c: Bootstrap starting.
I20260812 01:17:00.930370 25436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:00.931540 25436 log.cc:824] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:00.934602 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c: No bootstrap required, opened a new log
I20260812 01:17:00.935204 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:00.913242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:38639 (local address 127.0.0.1:41705)
0812 01:17:00.913989 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:00.914000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:00.914040 (+ 40us) server_negotiation.cc:407] Connection header received
0812 01:17:00.915750 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:00.915773 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:00.916184 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:00.916513 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:00.918456 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:00.919953 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:00.921716 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:00.922397 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:00.922526 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:00.927550 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:00.927559 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:00.927576 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:00.927620 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:00.930739 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:00.931701 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:00.931710 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:00.931719 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:00.931917 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:00.932443 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:00.932447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:00.932449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:00.933008 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:00.933328 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:00.933851 (+ 523us) server_negotiation.cc:299] Negotiation successful
0812 01:17:00.934157 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":309,"threads_started":1}
I20260812 01:17:00.939170 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:00.942816 25436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7600b7d9704961bd44afd87a045e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260812 01:17:00.943251 25436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:00.943398 25436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7600b7d9704961bd44afd87a045e9c, State: Initialized, Role: FOLLOWER
I20260812 01:17:00.944401 25436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7600b7d9704961bd44afd87a045e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260812 01:17:00.944712 25436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:00.944871 25436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:00.945048 25436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:00.947476 25436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7600b7d9704961bd44afd87a045e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260812 01:17:00.948019 25436 leader_election.cc:302] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d7600b7d9704961bd44afd87a045e9c; no voters:
I20260812 01:17:00.948669 25436 leader_election.cc:288] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:00.948858 25448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:00.951212 25436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:00.956051 25448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [term 1 LEADER]: Becoming Leader. State: Replica: 4d7600b7d9704961bd44afd87a045e9c, State: Running, Role: LEADER
I20260812 01:17:00.956725 25448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7600b7d9704961bd44afd87a045e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260812 01:17:00.959642 25449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d7600b7d9704961bd44afd87a045e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7600b7d9704961bd44afd87a045e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } } }
I20260812 01:17:00.960121 25449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:00.962440 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:00.943758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:36731 (local address 127.0.0.1:41705)
0812 01:17:00.944036 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:00.944046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:00.944362 (+ 316us) server_negotiation.cc:407] Connection header received
0812 01:17:00.944566 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:00.944570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:00.944637 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:00.944739 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:00.946542 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:00.947375 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:00.948868 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:00.949683 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:00.949811 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:00.953513 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:00.953521 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:00.953527 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:00.953580 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:00.957681 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:00.958790 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:00.958796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:00.958802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:00.959094 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:00.959779 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:00.959784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:00.959787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:00.960216 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:00.960561 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:00.961791 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0812 01:17:00.962005 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 01:17:00.968289 25466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:00.970271 25450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d7600b7d9704961bd44afd87a045e9c. Latest consensus state: current_term: 1 leader_uuid: "4d7600b7d9704961bd44afd87a045e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7600b7d9704961bd44afd87a045e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } } }
I20260812 01:17:00.970772 25450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:00.973819 25466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:00.983906 25466 catalog_manager.cc:1347] Generated new cluster ID: c9ada36f32ef45b3bc34c430886bb6c5
I20260812 01:17:00.984052 25466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:01.004655 25466 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:01.006654 25466 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:01.019028 25466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d7600b7d9704961bd44afd87a045e9c: Generated new TSK 0
I20260812 01:17:01.019946 25466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:01.031692 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25311
I20260812 01:17:01.032954 25429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:17:01.126020 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 25311
W20260812 01:17:01.149749 25285 connection.cc:535] client connection to 127.0.0.1:41705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41705): Connection reset by peer (error 104)
2026-08-12T01:17:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:01 dist-test-slave-21b9 krb5kdc[25491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:01 dist-test-slave-21b9 krb5kdc[25491](info): set up 2 sockets
Aug 12 01:17:01 dist-test-slave-21b9 krb5kdc[25491](info): commencing operation
krb5kdc: starting...
W20260812 01:17:02.446777 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.278s user 0.006s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:02 dist-test-slave-21b9 krb5kdc[25491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:17:02.831816 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:03.200115 25515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:03.200443 25515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:03.200594 25515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:03.207901 25515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:03.208050 25515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:03.208137 25515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:03.208221 25515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:03.208298 25515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:03.218390 25515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:03.219789 25515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:03.221124 25515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:03.232725 25524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:03.232877 25525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:03.233246 25529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:03.234053 25515 server_base.cc:1034] running on GCE node
I20260812 01:17:03.234876 25515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:03.236457 25515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:03.237700 25515 hybrid_clock.cc:648] HybridClock initialized: now 1786497423237649 us; error 38 us; skew 500 ppm
I20260812 01:17:03.238104 25515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:03 dist-test-slave-21b9 krb5kdc[25491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:03.241572 25515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:17:03.243314 25515 webserver.cc:460] Webserver started at http://127.0.0.1:39631/ using document root <none> and password file <none>
I20260812 01:17:03.244053 25515 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:03.244184 25515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:03.244503 25515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:03.247802 25515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "2eb317268ea84df699eee7880b989f68"
format_stamp: "Formatted at 2026-08-12 01:17:03 on dist-test-slave-21b9"
server_key: "04ba61713d0bf30131b61a8d8ff79a8c"
server_key_iv: "212867baea8387fdfa431235b8e043cf"
server_key_version: "encryptionkey@0"
I20260812 01:17:03.248625 25515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "2eb317268ea84df699eee7880b989f68"
format_stamp: "Formatted at 2026-08-12 01:17:03 on dist-test-slave-21b9"
server_key: "04ba61713d0bf30131b61a8d8ff79a8c"
server_key_iv: "212867baea8387fdfa431235b8e043cf"
server_key_version: "encryptionkey@0"
I20260812 01:17:03.254223 25515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 01:17:03.257843 25540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:03.259402 25515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:17:03.259588 25515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "2eb317268ea84df699eee7880b989f68"
format_stamp: "Formatted at 2026-08-12 01:17:03 on dist-test-slave-21b9"
server_key: "04ba61713d0bf30131b61a8d8ff79a8c"
server_key_iv: "212867baea8387fdfa431235b8e043cf"
server_key_version: "encryptionkey@0"
I20260812 01:17:03.259792 25515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:03.279590 25515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:03.284030 25515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:03.284322 25515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:03.303000 25515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34205
I20260812 01:17:03.303056 25643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34205 every 8 connection(s)
I20260812 01:17:03.304391 25515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:03.308283 25644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:03.314375 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 25515
I20260812 01:17:03.314579 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:03.315153 21084 external_mini_cluster.cc:1428] Setting key 2e904b5b1721d92b1b9c30a7a5ddb0a6
I20260812 01:17:03.320335 25644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68: Bootstrap starting.
I20260812 01:17:03.324194 25644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:03.325415 25644 log.cc:824] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:03.328708 25644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68: No bootstrap required, opened a new log
I20260812 01:17:03.329545 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:03.317162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:45819 (local address 127.0.0.1:34205)
0812 01:17:03.318107 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:03.318123 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:03.318166 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:17:03.319953 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:03.319978 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:03.320393 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:03.320757 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:03.321432 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:03.322821 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:03.324756 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:03.325555 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:03.325737 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:03.327040 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:03.327049 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:03.327065 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:03.327705 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:03.327790 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:03.327826 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:03.327975 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:03.328253 (+ 278us) server_negotiation.cc:299] Negotiation successful
0812 01:17:03.328428 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":462,"threads_started":1}
I20260812 01:17:03.334216 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:03.338718 25644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb317268ea84df699eee7880b989f68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34205 } }
I20260812 01:17:03.339200 25644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:03.339349 25644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb317268ea84df699eee7880b989f68, State: Initialized, Role: FOLLOWER
I20260812 01:17:03.340102 25644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb317268ea84df699eee7880b989f68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34205 } }
I20260812 01:17:03.340410 25644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:03.340580 25644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:03.340796 25644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:03.343739 25644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb317268ea84df699eee7880b989f68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34205 } }
I20260812 01:17:03.344306 25644 leader_election.cc:302] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eb317268ea84df699eee7880b989f68; no voters:
I20260812 01:17:03.344992 25644 leader_election.cc:288] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:03.345101 25659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:03.345981 25659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [term 1 LEADER]: Becoming Leader. State: Replica: 2eb317268ea84df699eee7880b989f68, State: Running, Role: LEADER
I20260812 01:17:03.346683 25659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb317268ea84df699eee7880b989f68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34205 } }
I20260812 01:17:03.347347 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:03.339732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:49235 (local address 127.0.0.1:34205)
0812 01:17:03.339943 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:03.339951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:03.340451 (+ 500us) server_negotiation.cc:407] Connection header received
0812 01:17:03.340758 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:03.340764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:03.340841 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:03.340957 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:03.341590 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:03.342757 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:03.344211 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:03.344966 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:03.345109 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:03.346383 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:03.346390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:03.346394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:03.346460 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:03.346543 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:03.346570 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:03.346701 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:03.346937 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 01:17:03.346991 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 01:17:03.347817 25644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:03.350360 25660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2eb317268ea84df699eee7880b989f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb317268ea84df699eee7880b989f68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34205 } } }
I20260812 01:17:03.350334 25661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eb317268ea84df699eee7880b989f68. Latest consensus state: current_term: 1 leader_uuid: "2eb317268ea84df699eee7880b989f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb317268ea84df699eee7880b989f68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34205 } } }
I20260812 01:17:03.350795 25661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:03.350791 25660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:03.352423 25667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:03.357527 25667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:03.366436 25667 catalog_manager.cc:1347] Generated new cluster ID: c655e33bc2794aadb9023ffa2aeca180
I20260812 01:17:03.366601 25667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:03.381233 25667 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:03.382586 25667 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:03.388343 25667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eb317268ea84df699eee7880b989f68: Generated new TSK 0
I20260812 01:17:03.389189 25667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:03.403519 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25515
W20260812 01:17:03.404723 25635 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.177:45819
W20260812 01:17:03.405511 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25515: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:17:03.405635 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 25515
W20260812 01:17:03.431600 25489 connection.cc:535] client connection to 127.0.0.1:34205 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34205): Connection reset by peer (error 104)
2026-08-12T01:17:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2281 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:03 dist-test-slave-21b9 krb5kdc[25704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:03 dist-test-slave-21b9 krb5kdc[25704](info): set up 2 sockets
Aug 12 01:17:03 dist-test-slave-21b9 krb5kdc[25704](info): commencing operation
krb5kdc: starting...
W20260812 01:17:04.715951 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:04 dist-test-slave-21b9 krb5kdc[25704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:17:05.106457 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:05.461200 25724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:05.461525 25724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:05.461668 25724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:05.469163 25724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:05.469295 25724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:05.469369 25724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:05.469457 25724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:05.469521 25724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:05.480238 25724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:05.481650 25724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:05.483016 25724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:05.495345 25734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:05.495373 25736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:05.495324 25740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:05.496600 25724 server_base.cc:1034] running on GCE node
I20260812 01:17:05.497474 25724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:05.498971 25724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:05.500172 25724 hybrid_clock.cc:648] HybridClock initialized: now 1786497425500121 us; error 45 us; skew 500 ppm
I20260812 01:17:05.500557 25724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:05 dist-test-slave-21b9 krb5kdc[25704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:05.503865 25724 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:17:05.505432 25724 webserver.cc:460] Webserver started at http://127.0.0.1:33055/ using document root <none> and password file <none>
I20260812 01:17:05.506141 25724 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:05.506251 25724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:05.506562 25724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:05.509711 25724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "c9752f2e33b64d74b45ec90d0f2225c1"
format_stamp: "Formatted at 2026-08-12 01:17:05 on dist-test-slave-21b9"
server_key: "8809427596c3314aa1a10dbf85052b10"
server_key_iv: "63e70e35c772f0350f416092718395cf"
server_key_version: "encryptionkey@0"
I20260812 01:17:05.510495 25724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "c9752f2e33b64d74b45ec90d0f2225c1"
format_stamp: "Formatted at 2026-08-12 01:17:05 on dist-test-slave-21b9"
server_key: "8809427596c3314aa1a10dbf85052b10"
server_key_iv: "63e70e35c772f0350f416092718395cf"
server_key_version: "encryptionkey@0"
I20260812 01:17:05.515889 25724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 01:17:05.519495 25758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:05.520838 25724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:17:05.521023 25724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "c9752f2e33b64d74b45ec90d0f2225c1"
format_stamp: "Formatted at 2026-08-12 01:17:05 on dist-test-slave-21b9"
server_key: "8809427596c3314aa1a10dbf85052b10"
server_key_iv: "63e70e35c772f0350f416092718395cf"
server_key_version: "encryptionkey@0"
I20260812 01:17:05.521219 25724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:05.527172 25724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:05.531569 25724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:05.531863 25724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:05.550316 25724 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43711
I20260812 01:17:05.550349 25852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43711 every 8 connection(s)
I20260812 01:17:05.551908 25724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:05.554430 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 25724
I20260812 01:17:05.554636 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:05.555178 21084 external_mini_cluster.cc:1428] Setting key a223685fbce91b608b8b2795af2f013a
I20260812 01:17:05.556428 25853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:05.564057 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:05.557082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:60045 (local address 127.0.0.1:43711)
0812 01:17:05.557938 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:05.557954 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:05.558008 (+ 54us) server_negotiation.cc:407] Connection header received
0812 01:17:05.560032 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:05.560062 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:05.560558 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:05.561004 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:05.562025 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:05.562034 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:05.562051 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:05.562068 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:05.562161 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:05.562188 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:05.562417 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:05.562561 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 01:17:05.562777 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":354,"threads_started":1}
I20260812 01:17:05.567821 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:05.571470 25853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1: Bootstrap starting.
I20260812 01:17:05.575675 25853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:05.575788 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:05.573241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:53899 (local address 127.0.0.1:43711)
0812 01:17:05.573453 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:05.573459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:05.573816 (+ 357us) server_negotiation.cc:407] Connection header received
0812 01:17:05.574022 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:05.574026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:05.574081 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:05.574166 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:05.575126 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:05.575130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:05.575133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:05.575148 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:05.575188 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:05.575202 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:05.575338 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:05.575438 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 01:17:05.575488 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 01:17:05.576831 25853 log.cc:824] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:05.579689 25853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1: No bootstrap required, opened a new log
I20260812 01:17:05.587113 25853 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9752f2e33b64d74b45ec90d0f2225c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260812 01:17:05.587641 25853 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:05.587740 25853 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9752f2e33b64d74b45ec90d0f2225c1, State: Initialized, Role: FOLLOWER
I20260812 01:17:05.588328 25853 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9752f2e33b64d74b45ec90d0f2225c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260812 01:17:05.588555 25853 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:05.588694 25853 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:05.588858 25853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:05.591112 25853 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9752f2e33b64d74b45ec90d0f2225c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260812 01:17:05.591665 25853 leader_election.cc:302] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9752f2e33b64d74b45ec90d0f2225c1; no voters:
I20260812 01:17:05.592301 25853 leader_election.cc:288] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:05.592449 25866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:05.593138 25866 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [term 1 LEADER]: Becoming Leader. State: Replica: c9752f2e33b64d74b45ec90d0f2225c1, State: Running, Role: LEADER
I20260812 01:17:05.593662 25866 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9752f2e33b64d74b45ec90d0f2225c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260812 01:17:05.594501 25853 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:05.596653 25869 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9752f2e33b64d74b45ec90d0f2225c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9752f2e33b64d74b45ec90d0f2225c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } } }
I20260812 01:17:05.596680 25870 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9752f2e33b64d74b45ec90d0f2225c1. Latest consensus state: current_term: 1 leader_uuid: "c9752f2e33b64d74b45ec90d0f2225c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9752f2e33b64d74b45ec90d0f2225c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } } }
I20260812 01:17:05.597126 25870 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:05.597126 25869 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:05.599110 25875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:05.604740 25875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:05.615227 25875 catalog_manager.cc:1347] Generated new cluster ID: d39eaf9371b243b594c3bb3fe74ba2bb
I20260812 01:17:05.615461 25875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:05.630857 25875 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:05.631409 25875 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:05.640250 25875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9752f2e33b64d74b45ec90d0f2225c1: Generated new TSK 0
I20260812 01:17:05.641352 25875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:05.656563 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25724
W20260812 01:17:05.658300 25843 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.177:60045
W20260812 01:17:05.659099 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:17:05.659240 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 25724
2026-08-12T01:17:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:05 dist-test-slave-21b9 krb5kdc[25898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:05 dist-test-slave-21b9 krb5kdc[25898](info): set up 2 sockets
Aug 12 01:17:05 dist-test-slave-21b9 krb5kdc[25898](info): commencing operation
krb5kdc: starting...
W20260812 01:17:06.972573 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.001s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:07 dist-test-slave-21b9 krb5kdc[25898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:17:07.356498 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:07.713757 25927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:07.714076 25927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:07.714214 25927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:07.721580 25927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:07.721724 25927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:07.721804 25927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:07.721885 25927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:07.721943 25927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:07.732226 25927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:07.733553 25927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:07.734931 25927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:07.746572 25934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:07.746641 25935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:07.746587 25939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:07.747830 25927 server_base.cc:1034] running on GCE node
I20260812 01:17:07.748678 25927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:07.750134 25927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:07.751379 25927 hybrid_clock.cc:648] HybridClock initialized: now 1786497427751314 us; error 45 us; skew 500 ppm
I20260812 01:17:07.751734 25927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:07 dist-test-slave-21b9 krb5kdc[25898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:07.754912 25927 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:17:07.756356 25927 webserver.cc:460] Webserver started at http://127.0.0.1:33857/ using document root <none> and password file <none>
I20260812 01:17:07.757015 25927 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:07.757119 25927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:07.757365 25927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:07.760419 25927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "b047004733a647c7bd6a319a400b08a6"
format_stamp: "Formatted at 2026-08-12 01:17:07 on dist-test-slave-21b9"
server_key: "918d8e6eddb6ddb4a4d1a6d3d6af5249"
server_key_iv: "a5931bf5f7d2056d1fdcdc0986e68918"
server_key_version: "encryptionkey@0"
I20260812 01:17:07.761168 25927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "b047004733a647c7bd6a319a400b08a6"
format_stamp: "Formatted at 2026-08-12 01:17:07 on dist-test-slave-21b9"
server_key: "918d8e6eddb6ddb4a4d1a6d3d6af5249"
server_key_iv: "a5931bf5f7d2056d1fdcdc0986e68918"
server_key_version: "encryptionkey@0"
I20260812 01:17:07.766498 25927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 01:17:07.770402 25949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:07.772248 25927 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 01:17:07.772414 25927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "b047004733a647c7bd6a319a400b08a6"
format_stamp: "Formatted at 2026-08-12 01:17:07 on dist-test-slave-21b9"
server_key: "918d8e6eddb6ddb4a4d1a6d3d6af5249"
server_key_iv: "a5931bf5f7d2056d1fdcdc0986e68918"
server_key_version: "encryptionkey@0"
I20260812 01:17:07.772607 25927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:07.796761 25927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:07.801090 25927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:07.801338 25927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:07.819952 25927 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38761
I20260812 01:17:07.819985 26046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38761 every 8 connection(s)
I20260812 01:17:07.821331 25927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:07.824440 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 25927
I20260812 01:17:07.824611 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:07.825075 21084 external_mini_cluster.cc:1428] Setting key bba7a444f79cf79e8efb8cf9fc857863
I20260812 01:17:07.825258 26047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:07.840070 26047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6: Bootstrap starting.
Aug 12 01:17:07 dist-test-slave-21b9 krb5kdc[25898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 01:17:07.844149 26047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:07.845608 26047 log.cc:824] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:07.849390 26047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6: No bootstrap required, opened a new log
I20260812 01:17:07.851130 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:07.826880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:44005 (local address 127.0.0.1:38761)
0812 01:17:07.827760 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:07.827776 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:07.827829 (+ 53us) server_negotiation.cc:407] Connection header received
0812 01:17:07.830187 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:07.830217 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:07.830761 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:07.831243 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:07.832635 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:07.834607 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:07.836031 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:07.837178 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:07.837379 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:07.841690 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:07.841704 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:07.841730 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:07.841783 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:07.845910 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:07.847075 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:07.847082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:07.847089 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:07.847367 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:07.847839 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:07.847844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:07.847847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:07.848576 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:07.848901 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:07.849389 (+ 488us) server_negotiation.cc:299] Negotiation successful
0812 01:17:07.849748 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":351,"threads_started":1}
I20260812 01:17:07.855394 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:07.858489 26047 raft_consensus.cc:348] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047004733a647c7bd6a319a400b08a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260812 01:17:07.858893 26047 raft_consensus.cc:374] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:07.859005 26047 raft_consensus.cc:729] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b047004733a647c7bd6a319a400b08a6, State: Initialized, Role: FOLLOWER
I20260812 01:17:07.859596 26047 consensus_queue.cc:260] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047004733a647c7bd6a319a400b08a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260812 01:17:07.859926 26047 raft_consensus.cc:388] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:07.860093 26047 raft_consensus.cc:482] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:07.860339 26047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:07.862969 26047 raft_consensus.cc:504] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047004733a647c7bd6a319a400b08a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260812 01:17:07.863647 26047 leader_election.cc:302] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b047004733a647c7bd6a319a400b08a6; no voters:
I20260812 01:17:07.864259 26047 leader_election.cc:288] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:07.864418 26066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:07.865312 26066 raft_consensus.cc:686] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [term 1 LEADER]: Becoming Leader. State: Replica: b047004733a647c7bd6a319a400b08a6, State: Running, Role: LEADER
I20260812 01:17:07.865944 26066 consensus_queue.cc:237] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047004733a647c7bd6a319a400b08a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260812 01:17:07.866756 26047 sys_catalog.cc:564] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:07.868793 26069 sys_catalog.cc:455] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b047004733a647c7bd6a319a400b08a6. Latest consensus state: current_term: 1 leader_uuid: "b047004733a647c7bd6a319a400b08a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047004733a647c7bd6a319a400b08a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } } }
I20260812 01:17:07.868912 26067 sys_catalog.cc:455] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b047004733a647c7bd6a319a400b08a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b047004733a647c7bd6a319a400b08a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } } }
I20260812 01:17:07.869211 26069 sys_catalog.cc:458] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:07.869241 26067 sys_catalog.cc:458] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:07.872470 26088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:07.877890 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:07.860010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:60423 (local address 127.0.0.1:38761)
0812 01:17:07.860213 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:07.860220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:07.860603 (+ 383us) server_negotiation.cc:407] Connection header received
0812 01:17:07.860731 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:07.860736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:07.860818 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:07.860923 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:07.862350 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:07.863460 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:07.865079 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:07.865749 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:07.865879 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:07.869528 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:07.869537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:07.869543 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:07.869594 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:07.873723 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:07.874837 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:07.874843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:07.874846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:07.875007 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:07.875682 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:07.875687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:07.875690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:07.876202 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:07.876712 (+ 510us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:07.877298 (+ 586us) server_negotiation.cc:299] Negotiation successful
0812 01:17:07.877510 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:17:07.878099 26088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:07.887457 26088 catalog_manager.cc:1347] Generated new cluster ID: d1e8690cc6bf4c6f87c5d992eb5fd27d
I20260812 01:17:07.887589 26088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:07.901248 26088 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:07.903092 26088 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:07.912055 26088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b047004733a647c7bd6a319a400b08a6: Generated new TSK 0
I20260812 01:17:07.913064 26088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:07.925717 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 25927
I20260812 01:17:07.927088 26041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:17:08.021556 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 25927
W20260812 01:17:08.044895 25896 connection.cc:535] client connection to 127.0.0.1:38761 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38761): Connection reset by peer (error 104)
2026-08-12T01:17:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:08 dist-test-slave-21b9 krb5kdc[26112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:08 dist-test-slave-21b9 krb5kdc[26112](info): set up 2 sockets
Aug 12 01:17:08 dist-test-slave-21b9 krb5kdc[26112](info): commencing operation
krb5kdc: starting...
W20260812 01:17:09.326718 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.000s sys 0.063s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:09 dist-test-slave-21b9 krb5kdc[26112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:17:09.696811 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:10.059044 26131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:10.059417 26131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:10.059602 26131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:10.067118 26131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:10.067248 26131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:10.067356 26131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:10.067432 26131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:10.067477 26131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:10.078137 26131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42119
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:10.079559 26131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:10.080904 26131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:10.092693 26142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:10.092767 26146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:10.092808 26143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:10.093962 26131 server_base.cc:1034] running on GCE node
I20260812 01:17:10.094784 26131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:10.096274 26131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:10.097472 26131 hybrid_clock.cc:648] HybridClock initialized: now 1786497430097423 us; error 43 us; skew 500 ppm
I20260812 01:17:10.097828 26131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:10 dist-test-slave-21b9 krb5kdc[26112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:10.101001 26131 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:17:10.102484 26131 webserver.cc:460] Webserver started at http://127.0.0.1:34059/ using document root <none> and password file <none>
I20260812 01:17:10.103217 26131 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:10.103365 26131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:10.103648 26131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:10.106635 26131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "c67d1fd69aa8479eb9dc9d98843aad2c"
format_stamp: "Formatted at 2026-08-12 01:17:10 on dist-test-slave-21b9"
server_key: "fc68e7bb89207480e7540ab261e65110"
server_key_iv: "d7a85fd4f47d70857462f8dc2e7765a2"
server_key_version: "encryptionkey@0"
I20260812 01:17:10.107437 26131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "c67d1fd69aa8479eb9dc9d98843aad2c"
format_stamp: "Formatted at 2026-08-12 01:17:10 on dist-test-slave-21b9"
server_key: "fc68e7bb89207480e7540ab261e65110"
server_key_iv: "d7a85fd4f47d70857462f8dc2e7765a2"
server_key_version: "encryptionkey@0"
I20260812 01:17:10.112639 26131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 01:17:10.116268 26154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:10.117681 26131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:17:10.117847 26131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "c67d1fd69aa8479eb9dc9d98843aad2c"
format_stamp: "Formatted at 2026-08-12 01:17:10 on dist-test-slave-21b9"
server_key: "fc68e7bb89207480e7540ab261e65110"
server_key_iv: "d7a85fd4f47d70857462f8dc2e7765a2"
server_key_version: "encryptionkey@0"
I20260812 01:17:10.118046 26131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:10.138120 26131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:10.142702 26131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:10.142992 26131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:10.162036 26131 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42119
I20260812 01:17:10.162062 26245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42119 every 8 connection(s)
I20260812 01:17:10.163546 26131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:10.164456 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 26131
I20260812 01:17:10.164673 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:10.165095 21084 external_mini_cluster.cc:1428] Setting key d642cd91a30a5eaacd7e20984bcc7b3a
I20260812 01:17:10.168200 26246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:10.178202 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:10.166969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:49647 (local address 127.0.0.1:42119)
0812 01:17:10.167781 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:10.167794 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:10.167837 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:17:10.169618 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:10.169642 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:10.170035 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:10.170405 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:10.170983 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:10.172381 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:10.173681 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:10.174339 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:10.174526 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:10.175914 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:10.175922 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:10.175936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:10.176495 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:10.176583 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:10.176608 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:10.176773 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:10.177083 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 01:17:10.177244 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":331,"threads_started":1}
I20260812 01:17:10.179482 26246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c: Bootstrap starting.
I20260812 01:17:10.182016 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:10.182937 26246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:10.184031 26246 log.cc:824] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:10.187220 26246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c: No bootstrap required, opened a new log
I20260812 01:17:10.192883 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:10.186702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:50187 (local address 127.0.0.1:42119)
0812 01:17:10.186845 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:10.186849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:10.187334 (+ 485us) server_negotiation.cc:407] Connection header received
0812 01:17:10.187519 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:10.187523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:10.187584 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:10.187669 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:10.188194 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:10.188993 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:10.190384 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:10.190855 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:10.190962 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:10.192166 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:10.192171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:10.192174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:10.192195 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:10.192232 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:10.192246 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:10.192346 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:10.192552 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 01:17:10.192609 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 01:17:10.196029 26246 raft_consensus.cc:348] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d1fd69aa8479eb9dc9d98843aad2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260812 01:17:10.196457 26246 raft_consensus.cc:374] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:10.196578 26246 raft_consensus.cc:729] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67d1fd69aa8479eb9dc9d98843aad2c, State: Initialized, Role: FOLLOWER
I20260812 01:17:10.197227 26246 consensus_queue.cc:260] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d1fd69aa8479eb9dc9d98843aad2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260812 01:17:10.197490 26246 raft_consensus.cc:388] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:10.197638 26246 raft_consensus.cc:482] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:10.197822 26246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:10.200577 26246 raft_consensus.cc:504] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d1fd69aa8479eb9dc9d98843aad2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260812 01:17:10.201165 26246 leader_election.cc:302] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67d1fd69aa8479eb9dc9d98843aad2c; no voters:
I20260812 01:17:10.201764 26246 leader_election.cc:288] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:10.201887 26269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:10.202594 26269 raft_consensus.cc:686] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [term 1 LEADER]: Becoming Leader. State: Replica: c67d1fd69aa8479eb9dc9d98843aad2c, State: Running, Role: LEADER
I20260812 01:17:10.203151 26269 consensus_queue.cc:237] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d1fd69aa8479eb9dc9d98843aad2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260812 01:17:10.204051 26246 sys_catalog.cc:564] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:10.206008 26270 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c67d1fd69aa8479eb9dc9d98843aad2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d1fd69aa8479eb9dc9d98843aad2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } } }
I20260812 01:17:10.206060 26273 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c67d1fd69aa8479eb9dc9d98843aad2c. Latest consensus state: current_term: 1 leader_uuid: "c67d1fd69aa8479eb9dc9d98843aad2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d1fd69aa8479eb9dc9d98843aad2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } } }
I20260812 01:17:10.206441 26270 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:10.206447 26273 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:10.209579 26280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:10.215339 26280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:10.225014 26280 catalog_manager.cc:1347] Generated new cluster ID: 4997cbaee87e4204a63d109c7fd2af44
I20260812 01:17:10.225126 26280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:10.241122 26280 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:10.242388 26280 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:10.249785 26280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c67d1fd69aa8479eb9dc9d98843aad2c: Generated new TSK 0
I20260812 01:17:10.250660 26280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:10.260794 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26131
W20260812 01:17:10.263002 26235 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.177:49647
W20260812 01:17:10.263813 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26131: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:17:10.263949 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 26131
W20260812 01:17:10.291548 26108 connection.cc:535] client connection to 127.0.0.1:42119 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42119): Connection reset by peer (error 104)
2026-08-12T01:17:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2246 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:10 dist-test-slave-21b9 krb5kdc[26307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:10 dist-test-slave-21b9 krb5kdc[26307](info): set up 2 sockets
Aug 12 01:17:10 dist-test-slave-21b9 krb5kdc[26307](info): commencing operation
krb5kdc: starting...
W20260812 01:17:11.564289 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.002s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:11 dist-test-slave-21b9 krb5kdc[26307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:17:11.934968 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:12.291168 26334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:12.291525 26334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:12.291675 26334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:12.299187 26334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:12.299353 26334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:12.299446 26334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:12.299525 26334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:12.299584 26334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:12.310019 26334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35451
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:12.311498 26334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:12.312842 26334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:12.324853 26347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:12.325078 26343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:12.324834 26342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:12.326009 26334 server_base.cc:1034] running on GCE node
I20260812 01:17:12.326922 26334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:12.328450 26334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:12.329680 26334 hybrid_clock.cc:648] HybridClock initialized: now 1786497432329632 us; error 39 us; skew 500 ppm
I20260812 01:17:12.330044 26334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:12 dist-test-slave-21b9 krb5kdc[26307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:12.333418 26334 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:17:12.335070 26334 webserver.cc:460] Webserver started at http://127.0.0.1:41733/ using document root <none> and password file <none>
I20260812 01:17:12.335842 26334 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:12.335973 26334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:12.336284 26334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:12.339453 26334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "3b71382c53314a77a8e094ce4d5f2c2a"
format_stamp: "Formatted at 2026-08-12 01:17:12 on dist-test-slave-21b9"
server_key: "e274eedcdab7198155bdc3da7115b647"
server_key_iv: "4575feb9e38d2613183855913a36f463"
server_key_version: "encryptionkey@0"
I20260812 01:17:12.340227 26334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "3b71382c53314a77a8e094ce4d5f2c2a"
format_stamp: "Formatted at 2026-08-12 01:17:12 on dist-test-slave-21b9"
server_key: "e274eedcdab7198155bdc3da7115b647"
server_key_iv: "4575feb9e38d2613183855913a36f463"
server_key_version: "encryptionkey@0"
I20260812 01:17:12.345631 26334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 01:17:12.349365 26357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:12.350817 26334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:17:12.350991 26334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "3b71382c53314a77a8e094ce4d5f2c2a"
format_stamp: "Formatted at 2026-08-12 01:17:12 on dist-test-slave-21b9"
server_key: "e274eedcdab7198155bdc3da7115b647"
server_key_iv: "4575feb9e38d2613183855913a36f463"
server_key_version: "encryptionkey@0"
I20260812 01:17:12.351186 26334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:12.357120 26334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:12.361838 26334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:12.362154 26334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:12.381043 26334 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35451
I20260812 01:17:12.381098 26429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35451 every 8 connection(s)
I20260812 01:17:12.382488 26334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:12.386366 26430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:12.392589 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 26334
I20260812 01:17:12.392778 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:12.393239 21084 external_mini_cluster.cc:1428] Setting key c85ec4f6f09d33ab7f97e9f05b3f9c6d
I20260812 01:17:12.398151 26430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a: Bootstrap starting.
I20260812 01:17:12.401631 26430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:12.401815 26441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:12.395659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:50051 (local address 127.0.0.1:35451)
0812 01:17:12.396507 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:12.396519 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:12.396569 (+ 50us) server_negotiation.cc:407] Connection header received
0812 01:17:12.398379 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:12.398404 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:12.398754 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:12.399110 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:12.400228 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:12.400236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:12.400248 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:12.400265 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:12.400329 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:12.400353 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:12.400567 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:12.400701 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 01:17:12.400883 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":370,"threads_started":1}
I20260812 01:17:12.402789 26430 log.cc:824] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:12.405640 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:12.406023 26430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a: No bootstrap required, opened a new log
I20260812 01:17:12.413444 26441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:12.410935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:53521 (local address 127.0.0.1:35451)
0812 01:17:12.411174 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:12.411181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:12.411551 (+ 370us) server_negotiation.cc:407] Connection header received
0812 01:17:12.411705 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:12.411709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:12.411760 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:12.411846 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:12.412806 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:12.412811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:12.412814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:12.412829 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:12.412873 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:12.412888 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:12.412987 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:12.413093 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 01:17:12.413147 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 01:17:12.415827 26430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71382c53314a77a8e094ce4d5f2c2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35451 } }
I20260812 01:17:12.416208 26430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:12.416303 26430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b71382c53314a77a8e094ce4d5f2c2a, State: Initialized, Role: FOLLOWER
I20260812 01:17:12.416975 26430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71382c53314a77a8e094ce4d5f2c2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35451 } }
I20260812 01:17:12.417201 26430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:12.417308 26430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:12.417450 26430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:12.419682 26430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71382c53314a77a8e094ce4d5f2c2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35451 } }
I20260812 01:17:12.420148 26430 leader_election.cc:302] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b71382c53314a77a8e094ce4d5f2c2a; no voters:
I20260812 01:17:12.420748 26430 leader_election.cc:288] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:12.420897 26453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:12.422086 26453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [term 1 LEADER]: Becoming Leader. State: Replica: 3b71382c53314a77a8e094ce4d5f2c2a, State: Running, Role: LEADER
I20260812 01:17:12.422659 26453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71382c53314a77a8e094ce4d5f2c2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35451 } }
I20260812 01:17:12.422973 26430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:12.426262 26455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b71382c53314a77a8e094ce4d5f2c2a. Latest consensus state: current_term: 1 leader_uuid: "3b71382c53314a77a8e094ce4d5f2c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71382c53314a77a8e094ce4d5f2c2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35451 } } }
I20260812 01:17:12.426710 26455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:12.426011 26454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b71382c53314a77a8e094ce4d5f2c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b71382c53314a77a8e094ce4d5f2c2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35451 } } }
I20260812 01:17:12.427139 26454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:12.429838 26467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:12.434213 26467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:12.442799 26467 catalog_manager.cc:1347] Generated new cluster ID: 4162486da19640daa0afb6baff5115ab
I20260812 01:17:12.443002 26467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:12.458166 26467 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:12.458635 26467 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:12.469758 26467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b71382c53314a77a8e094ce4d5f2c2a: Generated new TSK 0
I20260812 01:17:12.470690 26467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:12.481491 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26334
W20260812 01:17:12.482743 26423 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.177:50051
W20260812 01:17:12.483538 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26334: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:17:12.483685 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 26334
2026-08-12T01:17:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:12 dist-test-slave-21b9 krb5kdc[26483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:12 dist-test-slave-21b9 krb5kdc[26483](info): set up 2 sockets
Aug 12 01:17:12 dist-test-slave-21b9 krb5kdc[26483](info): commencing operation
krb5kdc: starting...
W20260812 01:17:13.780874 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.004s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:14 dist-test-slave-21b9 krb5kdc[26483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:17:14.167063 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:38093
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:38093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:14.524032 26506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:14.524381 26506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:14.524562 26506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:14.532325 26506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:14.532476 26506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:14.532567 26506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:14.532678 26506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:14.532747 26506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:14.543148 26506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:38093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:38093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:14.544592 26506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:14.546020 26506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:14.562915 26518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:14.563026 26516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:14.563045 26521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:14.564117 26506 server_base.cc:1034] running on GCE node
I20260812 01:17:14.564978 26506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:14.566459 26506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:14.567710 26506 hybrid_clock.cc:648] HybridClock initialized: now 1786497434567621 us; error 52 us; skew 500 ppm
I20260812 01:17:14.568127 26506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:14 dist-test-slave-21b9 krb5kdc[26483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497434, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:14.575709 26506 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:17:14.577433 26506 webserver.cc:460] Webserver started at http://127.20.151.62:37541/ using document root <none> and password file <none>
I20260812 01:17:14.578212 26506 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:14.578354 26506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:14.578686 26506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:14.582108 26506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "ebfabe3c276a49d2a03ee4a3b0856f30"
format_stamp: "Formatted at 2026-08-12 01:17:14 on dist-test-slave-21b9"
server_key: "6962ffa63f9888752b4ec3c645031f35"
server_key_iv: "e9394ce015cf85e3219187ce56659e79"
server_key_version: "encryptionkey@0"
I20260812 01:17:14.582957 26506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "ebfabe3c276a49d2a03ee4a3b0856f30"
format_stamp: "Formatted at 2026-08-12 01:17:14 on dist-test-slave-21b9"
server_key: "6962ffa63f9888752b4ec3c645031f35"
server_key_iv: "e9394ce015cf85e3219187ce56659e79"
server_key_version: "encryptionkey@0"
I20260812 01:17:14.588682 26506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 01:17:14.592360 26531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:14.593761 26506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:17:14.593976 26506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "ebfabe3c276a49d2a03ee4a3b0856f30"
format_stamp: "Formatted at 2026-08-12 01:17:14 on dist-test-slave-21b9"
server_key: "6962ffa63f9888752b4ec3c645031f35"
server_key_iv: "e9394ce015cf85e3219187ce56659e79"
server_key_version: "encryptionkey@0"
I20260812 01:17:14.594187 26506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:14.619848 26506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:14.624207 26506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:14.624522 26506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:14.644739 26506 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:38093
I20260812 01:17:14.644773 26622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:38093 every 8 connection(s)
I20260812 01:17:14.646199 26506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:14.650128 26625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:14.654563 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 26506
I20260812 01:17:14.654757 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:14.655355 21084 external_mini_cluster.cc:1428] Setting key 4348d58c15b2a25f0164e9ec6f29351f
I20260812 01:17:14.661959 26625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30: Bootstrap starting.
I20260812 01:17:14.665174 26625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:14.666198 26625 log.cc:824] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:14.669749 26625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30: No bootstrap required, opened a new log
Aug 12 01:17:14 dist-test-slave-21b9 krb5kdc[26483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:17:14.679139 26625 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfabe3c276a49d2a03ee4a3b0856f30" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38093 } }
I20260812 01:17:14.679600 26625 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:14.679704 26625 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfabe3c276a49d2a03ee4a3b0856f30, State: Initialized, Role: FOLLOWER
I20260812 01:17:14.680135 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:14.657421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46253 (local address 127.20.151.62:38093)
0812 01:17:14.658260 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:14.658272 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:14.658318 (+ 46us) server_negotiation.cc:407] Connection header received
0812 01:17:14.660786 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:14.660813 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:14.661221 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:14.661602 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:14.663374 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:14.664785 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:14.666168 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:14.667110 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:14.667355 (+ 245us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:14.672322 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:14.672330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:14.672349 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:14.672385 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:14.675575 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:14.676548 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:14.676555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:14.676561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:14.676734 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:14.677263 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:14.677267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:14.677269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:14.677863 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:14.678132 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:14.678647 (+ 515us) server_negotiation.cc:299] Negotiation successful
0812 01:17:14.678934 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":351,"threads_started":1}
I20260812 01:17:14.680325 26625 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfabe3c276a49d2a03ee4a3b0856f30" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38093 } }
I20260812 01:17:14.680588 26625 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:14.680738 26625 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:14.680923 26625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:14.683486 26625 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfabe3c276a49d2a03ee4a3b0856f30" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38093 } }
I20260812 01:17:14.684051 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:14.683987 26625 leader_election.cc:302] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebfabe3c276a49d2a03ee4a3b0856f30; no voters:
I20260812 01:17:14.684690 26625 leader_election.cc:288] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:14.684785 26637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:14.686784 26625 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:14.692312 26637 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [term 1 LEADER]: Becoming Leader. State: Replica: ebfabe3c276a49d2a03ee4a3b0856f30, State: Running, Role: LEADER
I20260812 01:17:14.693141 26637 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfabe3c276a49d2a03ee4a3b0856f30" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38093 } }
I20260812 01:17:14.696230 26639 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebfabe3c276a49d2a03ee4a3b0856f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfabe3c276a49d2a03ee4a3b0856f30" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38093 } } }
I20260812 01:17:14.696266 26640 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebfabe3c276a49d2a03ee4a3b0856f30. Latest consensus state: current_term: 1 leader_uuid: "ebfabe3c276a49d2a03ee4a3b0856f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfabe3c276a49d2a03ee4a3b0856f30" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 38093 } } }
I20260812 01:17:14.696718 26639 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:14.696720 26640 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:14.699240 26660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:14.704669 26660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:14.712788 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:14.693130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42369 (local address 127.20.151.62:38093)
0812 01:17:14.693333 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:14.693340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:14.693863 (+ 523us) server_negotiation.cc:407] Connection header received
0812 01:17:14.694072 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:14.694077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:14.694158 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:14.694267 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:14.698580 (+ 4313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:14.699810 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:14.701590 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:14.702079 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:14.702192 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:14.705371 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:14.705380 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:14.705386 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:14.705430 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:14.709560 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:14.710277 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:14.710283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:14.710291 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:14.710425 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:14.710944 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:14.710949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:14.710952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:14.711437 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:14.711719 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:14.712310 (+ 591us) server_negotiation.cc:299] Negotiation successful
0812 01:17:14.712493 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:17:14.717667 26660 catalog_manager.cc:1347] Generated new cluster ID: a0c1614872c94348b793e6554a75421f
I20260812 01:17:14.717783 26660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:14.755223 26660 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:14.756595 26660 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:14.768407 26660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebfabe3c276a49d2a03ee4a3b0856f30: Generated new TSK 0
I20260812 01:17:14.769277 26660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:14.774684 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26506
I20260812 01:17:14.775938 26617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:17:14.871776 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 26506
W20260812 01:17:14.898412 26476 connection.cc:535] client connection to 127.20.151.62:38093 recv error: Network error: failed to read from TLS socket (remote: 127.20.151.62:38093): Connection reset by peer (error 104)
2026-08-12T01:17:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:15 dist-test-slave-21b9 krb5kdc[26690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:15 dist-test-slave-21b9 krb5kdc[26690](info): set up 2 sockets
Aug 12 01:17:15 dist-test-slave-21b9 krb5kdc[26690](info): commencing operation
krb5kdc: starting...
W20260812 01:17:16.181850 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.264s user 0.001s sys 0.063s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:16 dist-test-slave-21b9 krb5kdc[26690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:17:16.567018 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43947
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:43947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:16.918605 26712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:16.918932 26712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:16.919097 26712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:16.926424 26712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:16.926574 26712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:16.926662 26712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:16.926753 26712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:16.926820 26712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:16.937013 26712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:43947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:16.938338 26712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:16.939761 26712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:16.951834 26718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:16.951793 26722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:16.951753 26719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:16.952670 26712 server_base.cc:1034] running on GCE node
I20260812 01:17:16.953523 26712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:16.955005 26712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:16.956212 26712 hybrid_clock.cc:648] HybridClock initialized: now 1786497436956169 us; error 32 us; skew 500 ppm
I20260812 01:17:16.956601 26712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:16 dist-test-slave-21b9 krb5kdc[26690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497436, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:16.959888 26712 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:17:16.961403 26712 webserver.cc:460] Webserver started at http://127.20.151.62:35373/ using document root <none> and password file <none>
I20260812 01:17:16.962095 26712 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:16.962212 26712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:16.962509 26712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:16.965691 26712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "c8d11bdf4f6b442f98aea0fcc78d6ff1"
format_stamp: "Formatted at 2026-08-12 01:17:16 on dist-test-slave-21b9"
server_key: "371a4f966fbd00463a3b092ff2ddd0a5"
server_key_iv: "caee8b889967f57eafc441016b1a6d86"
server_key_version: "encryptionkey@0"
I20260812 01:17:16.966470 26712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "c8d11bdf4f6b442f98aea0fcc78d6ff1"
format_stamp: "Formatted at 2026-08-12 01:17:16 on dist-test-slave-21b9"
server_key: "371a4f966fbd00463a3b092ff2ddd0a5"
server_key_iv: "caee8b889967f57eafc441016b1a6d86"
server_key_version: "encryptionkey@0"
I20260812 01:17:16.972050 26712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 01:17:16.975618 26732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:16.976977 26712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:17:16.977154 26712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "c8d11bdf4f6b442f98aea0fcc78d6ff1"
format_stamp: "Formatted at 2026-08-12 01:17:16 on dist-test-slave-21b9"
server_key: "371a4f966fbd00463a3b092ff2ddd0a5"
server_key_iv: "caee8b889967f57eafc441016b1a6d86"
server_key_version: "encryptionkey@0"
I20260812 01:17:16.977345 26712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:17.002408 26712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:17.006991 26712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:17.007333 26712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:17.025987 26712 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:43947
I20260812 01:17:17.026008 26824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:43947 every 8 connection(s)
I20260812 01:17:17.027432 26712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:17.031302 26825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:17.034145 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 26712
I20260812 01:17:17.034343 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:17.034885 21084 external_mini_cluster.cc:1428] Setting key 1d3065bc45972a6c10112305d8f7fa8f
I20260812 01:17:17.043560 26825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1: Bootstrap starting.
I20260812 01:17:17.047147 26825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:17.048381 26825 log.cc:824] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:17.048790 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:17.036808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58167 (local address 127.20.151.62:43947)
0812 01:17:17.037610 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:17.037622 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:17.037666 (+ 44us) server_negotiation.cc:407] Connection header received
0812 01:17:17.039417 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:17.039441 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:17.039868 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:17.040210 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:17.040867 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:17.042255 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:17.044179 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:17.044862 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:17.044997 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:17.046347 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:17.046356 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:17.046370 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:17.046941 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:17.047021 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:17.047057 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:17.047182 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:17.047517 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 01:17:17.047718 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":364,"threads_started":1}
I20260812 01:17:17.051668 26825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1: No bootstrap required, opened a new log
I20260812 01:17:17.052968 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:17.061964 26825 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d11bdf4f6b442f98aea0fcc78d6ff1" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43947 } }
I20260812 01:17:17.062392 26825 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:17.062564 26825 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d11bdf4f6b442f98aea0fcc78d6ff1, State: Initialized, Role: FOLLOWER
I20260812 01:17:17.063180 26825 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d11bdf4f6b442f98aea0fcc78d6ff1" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43947 } }
I20260812 01:17:17.063465 26825 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:17.063601 26825 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:17.063766 26825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:17.065151 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:17.058342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50411 (local address 127.20.151.62:43947)
0812 01:17:17.058550 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:17.058556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:17.058991 (+ 435us) server_negotiation.cc:407] Connection header received
0812 01:17:17.059156 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:17.059161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:17.059252 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:17.059396 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:17.060009 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:17.061152 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:17.062509 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:17.063079 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:17.063204 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:17.064394 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:17.064400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:17.064406 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:17.064425 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:17.064486 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:17.064503 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:17.064628 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:17.064807 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 01:17:17.064863 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:17:17.066401 26825 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d11bdf4f6b442f98aea0fcc78d6ff1" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43947 } }
I20260812 01:17:17.067003 26825 leader_election.cc:302] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8d11bdf4f6b442f98aea0fcc78d6ff1; no voters:
I20260812 01:17:17.067622 26825 leader_election.cc:288] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:17.067745 26841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:17.068483 26841 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [term 1 LEADER]: Becoming Leader. State: Replica: c8d11bdf4f6b442f98aea0fcc78d6ff1, State: Running, Role: LEADER
I20260812 01:17:17.069016 26841 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d11bdf4f6b442f98aea0fcc78d6ff1" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43947 } }
I20260812 01:17:17.069865 26825 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:17.072161 26843 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8d11bdf4f6b442f98aea0fcc78d6ff1. Latest consensus state: current_term: 1 leader_uuid: "c8d11bdf4f6b442f98aea0fcc78d6ff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d11bdf4f6b442f98aea0fcc78d6ff1" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43947 } } }
I20260812 01:17:17.072191 26842 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8d11bdf4f6b442f98aea0fcc78d6ff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d11bdf4f6b442f98aea0fcc78d6ff1" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43947 } } }
I20260812 01:17:17.072724 26842 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:17.072721 26843 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:17.074436 26847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:17.079975 26847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:17.091089 26847 catalog_manager.cc:1347] Generated new cluster ID: 4dbf049c2b8345bfbab58771e8c63447
I20260812 01:17:17.091223 26847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:17.113927 26847 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:17.115341 26847 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:17.124596 26847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8d11bdf4f6b442f98aea0fcc78d6ff1: Generated new TSK 0
I20260812 01:17:17.125418 26847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:17.131691 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26712
W20260812 01:17:17.132889 26813 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58167
W20260812 01:17:17.133700 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26712: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:17:17.133833 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 26712
W20260812 01:17:17.160310 26683 connection.cc:535] client connection to 127.20.151.62:43947 recv error: Network error: failed to read from TLS socket (remote: 127.20.151.62:43947): Connection reset by peer (error 104)
2026-08-12T01:17:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2261 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:17 dist-test-slave-21b9 krb5kdc[26880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:17 dist-test-slave-21b9 krb5kdc[26880](info): set up 2 sockets
Aug 12 01:17:17 dist-test-slave-21b9 krb5kdc[26880](info): commencing operation
krb5kdc: starting...
W20260812 01:17:18.438117 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.000s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:18 dist-test-slave-21b9 krb5kdc[26880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:17:18.810599 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43425
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:43425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:19.169818 26904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:19.170168 26904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:19.170321 26904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:19.177855 26904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:19.178004 26904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:19.178090 26904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:19.178184 26904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:19.178243 26904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:19.188644 26904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:43425
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:19.190039 26904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:19.191422 26904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:19.203336 26918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:19.203393 26913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:19.203357 26912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:19.204540 26904 server_base.cc:1034] running on GCE node
I20260812 01:17:19.205385 26904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:19.206851 26904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:19.208076 26904 hybrid_clock.cc:648] HybridClock initialized: now 1786497439208000 us; error 55 us; skew 500 ppm
I20260812 01:17:19.208432 26904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:19 dist-test-slave-21b9 krb5kdc[26880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497439, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:19.211625 26904 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:17:19.213091 26904 webserver.cc:460] Webserver started at http://127.20.151.62:40139/ using document root <none> and password file <none>
I20260812 01:17:19.213790 26904 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:19.213915 26904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:19.214236 26904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:19.217679 26904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "a8fe28fdee54403998d4608b27a56e1b"
format_stamp: "Formatted at 2026-08-12 01:17:19 on dist-test-slave-21b9"
server_key: "390da80c23ee0e7435aa4e971e7df7dd"
server_key_iv: "d772d8bafb7b0104c1a42ad9742cda5e"
server_key_version: "encryptionkey@0"
I20260812 01:17:19.218489 26904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "a8fe28fdee54403998d4608b27a56e1b"
format_stamp: "Formatted at 2026-08-12 01:17:19 on dist-test-slave-21b9"
server_key: "390da80c23ee0e7435aa4e971e7df7dd"
server_key_iv: "d772d8bafb7b0104c1a42ad9742cda5e"
server_key_version: "encryptionkey@0"
I20260812 01:17:19.224227 26904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:17:19.227959 26930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:19.229377 26904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:17:19.229562 26904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "a8fe28fdee54403998d4608b27a56e1b"
format_stamp: "Formatted at 2026-08-12 01:17:19 on dist-test-slave-21b9"
server_key: "390da80c23ee0e7435aa4e971e7df7dd"
server_key_iv: "d772d8bafb7b0104c1a42ad9742cda5e"
server_key_version: "encryptionkey@0"
I20260812 01:17:19.229765 26904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:19.235670 26904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:19.239640 26904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:19.239914 26904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:19.257328 26904 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:43425
I20260812 01:17:19.257356 27014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:43425 every 8 connection(s)
I20260812 01:17:19.258757 26904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:19.262457 27015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:19.267509 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 26904
I20260812 01:17:19.267688 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:19.268105 21084 external_mini_cluster.cc:1428] Setting key 1327822609c4245e1f8064bd3457ddf7
I20260812 01:17:19.274984 27015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b: Bootstrap starting.
I20260812 01:17:19.277110 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:19.269847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58971 (local address 127.20.151.62:43425)
0812 01:17:19.270661 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:19.270680 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:19.270731 (+ 51us) server_negotiation.cc:407] Connection header received
0812 01:17:19.273228 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:19.273258 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:19.273712 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:19.274148 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:19.275155 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:19.275164 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:19.275179 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:19.275197 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:19.275264 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:19.275341 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:19.275566 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:19.275698 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 01:17:19.275885 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":310,"threads_started":1}
I20260812 01:17:19.278885 27015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:19.280177 27015 log.cc:824] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:19.281162 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:19.283574 27015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b: No bootstrap required, opened a new log
I20260812 01:17:19.289021 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:19.286573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53073 (local address 127.20.151.62:43425)
0812 01:17:19.286760 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:19.286766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:19.287030 (+ 264us) server_negotiation.cc:407] Connection header received
0812 01:17:19.287171 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:19.287176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:19.287238 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:19.287375 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:19.288269 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:19.288278 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:19.288283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:19.288302 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:19.288353 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:19.288375 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:19.288505 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:19.288643 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 01:17:19.288702 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:17:19.293033 27015 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fe28fdee54403998d4608b27a56e1b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43425 } }
I20260812 01:17:19.293387 27015 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:19.293462 27015 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fe28fdee54403998d4608b27a56e1b, State: Initialized, Role: FOLLOWER
I20260812 01:17:19.294008 27015 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fe28fdee54403998d4608b27a56e1b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43425 } }
I20260812 01:17:19.294219 27015 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:19.294324 27015 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:19.294458 27015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:19.296713 27015 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fe28fdee54403998d4608b27a56e1b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43425 } }
I20260812 01:17:19.297278 27015 leader_election.cc:302] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8fe28fdee54403998d4608b27a56e1b; no voters:
I20260812 01:17:19.297957 27015 leader_election.cc:288] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:19.298049 27036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:19.298710 27036 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [term 1 LEADER]: Becoming Leader. State: Replica: a8fe28fdee54403998d4608b27a56e1b, State: Running, Role: LEADER
I20260812 01:17:19.299363 27036 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fe28fdee54403998d4608b27a56e1b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43425 } }
I20260812 01:17:19.300052 27015 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:19.302320 27037 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8fe28fdee54403998d4608b27a56e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fe28fdee54403998d4608b27a56e1b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43425 } } }
I20260812 01:17:19.302399 27038 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8fe28fdee54403998d4608b27a56e1b. Latest consensus state: current_term: 1 leader_uuid: "a8fe28fdee54403998d4608b27a56e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fe28fdee54403998d4608b27a56e1b" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43425 } } }
I20260812 01:17:19.302721 27037 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:19.302744 27038 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:19.310019 27045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:19.315498 27045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:19.324396 27045 catalog_manager.cc:1347] Generated new cluster ID: cce6d2acead3453999a923a56a7ac234
I20260812 01:17:19.324534 27045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:19.338860 27045 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:19.339336 27045 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:19.345407 27045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8fe28fdee54403998d4608b27a56e1b: Generated new TSK 0
I20260812 01:17:19.346462 27045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:19.361934 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26904
W20260812 01:17:19.363058 27003 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58971
W20260812 01:17:19.363796 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 26904: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:17:19.363968 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 26904
2026-08-12T01:17:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:19 dist-test-slave-21b9 krb5kdc[27076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:19 dist-test-slave-21b9 krb5kdc[27076](info): set up 2 sockets
Aug 12 01:17:19 dist-test-slave-21b9 krb5kdc[27076](info): commencing operation
krb5kdc: starting...
W20260812 01:17:20.665827 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.001s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:20 dist-test-slave-21b9 krb5kdc[27076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:17:21.042198 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43107
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:43107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:21.396551 27095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:21.396864 27095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:21.396996 27095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:21.404278 27095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:21.404397 27095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:21.404449 27095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:21.404511 27095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:21.404556 27095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:21.414572 27095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:43107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:43107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:21.415872 27095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:21.417115 27095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:21.428531 27107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:21.428633 27108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:21.428663 27112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:21.429553 27095 server_base.cc:1034] running on GCE node
I20260812 01:17:21.430361 27095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:21.431756 27095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:21.432966 27095 hybrid_clock.cc:648] HybridClock initialized: now 1786497441432924 us; error 30 us; skew 500 ppm
I20260812 01:17:21.433318 27095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:21 dist-test-slave-21b9 krb5kdc[27076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497441, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:21.439895 27095 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:17:21.441375 27095 webserver.cc:460] Webserver started at http://127.20.151.62:44929/ using document root <none> and password file <none>
I20260812 01:17:21.442052 27095 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:21.442183 27095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:21.442476 27095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:21.445554 27095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "f25b18dcaff04fc2bc28780f9eb13159"
format_stamp: "Formatted at 2026-08-12 01:17:21 on dist-test-slave-21b9"
server_key: "78de7cfb5db18787610cfa8ee884797d"
server_key_iv: "617e03d19f97a6553de644d913572218"
server_key_version: "encryptionkey@0"
I20260812 01:17:21.446318 27095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "f25b18dcaff04fc2bc28780f9eb13159"
format_stamp: "Formatted at 2026-08-12 01:17:21 on dist-test-slave-21b9"
server_key: "78de7cfb5db18787610cfa8ee884797d"
server_key_iv: "617e03d19f97a6553de644d913572218"
server_key_version: "encryptionkey@0"
I20260812 01:17:21.451681 27095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:17:21.455101 27121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:21.456418 27095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:17:21.456590 27095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "f25b18dcaff04fc2bc28780f9eb13159"
format_stamp: "Formatted at 2026-08-12 01:17:21 on dist-test-slave-21b9"
server_key: "78de7cfb5db18787610cfa8ee884797d"
server_key_iv: "617e03d19f97a6553de644d913572218"
server_key_version: "encryptionkey@0"
I20260812 01:17:21.456774 27095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:21.475518 27095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:21.479912 27095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:21.480198 27095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:21.498886 27095 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:43107
I20260812 01:17:21.498893 27202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:43107 every 8 connection(s)
I20260812 01:17:21.500367 27095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:21.504202 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:21.509205 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 27095
I20260812 01:17:21.509407 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:21.509966 21084 external_mini_cluster.cc:1428] Setting key 52f456d1779badad4b26d0a4c2ae5357
I20260812 01:17:21.515802 27204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159: Bootstrap starting.
I20260812 01:17:21.519404 27204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:21.520435 27204 log.cc:824] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:21.523939 27204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159: No bootstrap required, opened a new log
Aug 12 01:17:21 dist-test-slave-21b9 krb5kdc[27076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:17:21.531840 27204 raft_consensus.cc:348] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b18dcaff04fc2bc28780f9eb13159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43107 } }
I20260812 01:17:21.532280 27204 raft_consensus.cc:374] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:21.532404 27204 raft_consensus.cc:729] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25b18dcaff04fc2bc28780f9eb13159, State: Initialized, Role: FOLLOWER
I20260812 01:17:21.532989 27204 consensus_queue.cc:260] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b18dcaff04fc2bc28780f9eb13159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43107 } }
I20260812 01:17:21.533221 27204 raft_consensus.cc:388] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:21.533360 27204 raft_consensus.cc:482] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:21.533529 27204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:21.533564 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:21.511960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57353 (local address 127.20.151.62:43107)
0812 01:17:21.512817 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:21.512828 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:21.512875 (+ 47us) server_negotiation.cc:407] Connection header received
0812 01:17:21.514650 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:21.514673 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:21.515072 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:21.515463 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:21.516961 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:21.518782 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:21.520128 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:21.520865 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:21.521085 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:21.525893 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:21.525903 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:21.525927 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:21.525973 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:21.529164 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:21.529871 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:21.529881 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:21.529888 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:21.530079 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:21.530480 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:21.530488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:21.530490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:21.531091 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:21.531401 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:21.531895 (+ 494us) server_negotiation.cc:299] Negotiation successful
0812 01:17:21.532352 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":402,"threads_started":1}
I20260812 01:17:21.536118 27204 raft_consensus.cc:504] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b18dcaff04fc2bc28780f9eb13159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43107 } }
I20260812 01:17:21.536644 27204 leader_election.cc:302] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f25b18dcaff04fc2bc28780f9eb13159; no voters:
I20260812 01:17:21.537288 27204 leader_election.cc:288] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:21.537406 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:21.537405 27220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:21.539381 27204 sys_catalog.cc:564] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:21.539554 27220 raft_consensus.cc:686] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [term 1 LEADER]: Becoming Leader. State: Replica: f25b18dcaff04fc2bc28780f9eb13159, State: Running, Role: LEADER
I20260812 01:17:21.540098 27220 consensus_queue.cc:237] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b18dcaff04fc2bc28780f9eb13159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43107 } }
I20260812 01:17:21.542654 27222 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f25b18dcaff04fc2bc28780f9eb13159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b18dcaff04fc2bc28780f9eb13159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43107 } } }
I20260812 01:17:21.543077 27222 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:21.543612 27225 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f25b18dcaff04fc2bc28780f9eb13159. Latest consensus state: current_term: 1 leader_uuid: "f25b18dcaff04fc2bc28780f9eb13159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b18dcaff04fc2bc28780f9eb13159" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 43107 } } }
I20260812 01:17:21.543890 27225 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:21.547474 27231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:21.554051 27231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:21.569548 27231 catalog_manager.cc:1347] Generated new cluster ID: a0fcfbd89a0146ed9524719799e52459
I20260812 01:17:21.569684 27231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:21.577951 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:21.549828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54871 (local address 127.20.151.62:43107)
0812 01:17:21.550010 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:21.550016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:21.559121 (+ 9105us) server_negotiation.cc:407] Connection header received
0812 01:17:21.559333 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:21.559339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:21.559471 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:21.559596 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:21.562002 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:21.563146 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:21.565074 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:21.565811 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:21.565933 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:21.570026 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:21.570033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:21.570040 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:21.570091 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:21.574174 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:21.575177 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:21.575184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:21.575188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:21.575383 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:21.575861 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:21.575866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:21.575870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:21.576335 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:21.576567 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:21.577236 (+ 669us) server_negotiation.cc:299] Negotiation successful
0812 01:17:21.577508 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:17:21.598165 27231 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:21.599937 27231 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:21.610879 27231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f25b18dcaff04fc2bc28780f9eb13159: Generated new TSK 0
I20260812 01:17:21.611871 27231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:21.628871 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27095
I20260812 01:17:21.630236 27197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:17:21.724788 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 27095
2026-08-12T01:17:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:21 dist-test-slave-21b9 krb5kdc[27275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:21 dist-test-slave-21b9 krb5kdc[27275](info): set up 2 sockets
Aug 12 01:17:21 dist-test-slave-21b9 krb5kdc[27275](info): commencing operation
krb5kdc: starting...
W20260812 01:17:23.030705 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.002s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:23 dist-test-slave-21b9 krb5kdc[27275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:17:23.400024 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:41721
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:41721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:23.744557 27292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:23.744856 27292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:23.744992 27292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:23.752275 27292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:23.752399 27292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:23.752481 27292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:23.752578 27292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:23.752631 27292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:23.762567 27292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:41721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:41721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:23.763901 27292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:23.765251 27292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:23.777004 27301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:23.777040 27302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:23.777056 27304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:23.777912 27292 server_base.cc:1034] running on GCE node
I20260812 01:17:23.778714 27292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:23.780228 27292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:23.781453 27292 hybrid_clock.cc:648] HybridClock initialized: now 1786497443781407 us; error 29 us; skew 500 ppm
I20260812 01:17:23.781811 27292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:23 dist-test-slave-21b9 krb5kdc[27275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497443, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:23.784883 27292 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:17:23.786384 27292 webserver.cc:460] Webserver started at http://127.20.151.62:39329/ using document root <none> and password file <none>
I20260812 01:17:23.787055 27292 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:23.787184 27292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:23.787505 27292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:23.790536 27292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "6ab3712acacd4167a10203843d852e0e"
format_stamp: "Formatted at 2026-08-12 01:17:23 on dist-test-slave-21b9"
server_key: "4e34964e7eb785558ba05ccf2bb43f4c"
server_key_iv: "1b8bcd16320ac87d2b99f1834226dd35"
server_key_version: "encryptionkey@0"
I20260812 01:17:23.791316 27292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "6ab3712acacd4167a10203843d852e0e"
format_stamp: "Formatted at 2026-08-12 01:17:23 on dist-test-slave-21b9"
server_key: "4e34964e7eb785558ba05ccf2bb43f4c"
server_key_iv: "1b8bcd16320ac87d2b99f1834226dd35"
server_key_version: "encryptionkey@0"
I20260812 01:17:23.796382 27292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 01:17:23.799741 27316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:23.800984 27292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 01:17:23.801160 27292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "6ab3712acacd4167a10203843d852e0e"
format_stamp: "Formatted at 2026-08-12 01:17:23 on dist-test-slave-21b9"
server_key: "4e34964e7eb785558ba05ccf2bb43f4c"
server_key_iv: "1b8bcd16320ac87d2b99f1834226dd35"
server_key_version: "encryptionkey@0"
I20260812 01:17:23.801342 27292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:23.841363 27292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:23.845634 27292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:23.845909 27292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:23.864066 27292 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:41721
I20260812 01:17:23.864091 27392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:41721 every 8 connection(s)
I20260812 01:17:23.865425 27292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:23.869159 27393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:23.875232 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 27292
I20260812 01:17:23.875420 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:23.875818 21084 external_mini_cluster.cc:1428] Setting key 641ebc64549daf7fa18a76e5019e1566
I20260812 01:17:23.880105 27393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e: Bootstrap starting.
I20260812 01:17:23.883584 27393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:23.884833 27393 log.cc:824] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:23.887948 27393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e: No bootstrap required, opened a new log
I20260812 01:17:23.888814 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:23.877574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38115 (local address 127.20.151.62:41721)
0812 01:17:23.878428 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:23.878439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:23.878482 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:17:23.880215 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:23.880244 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:23.880620 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:23.880916 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:23.881507 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:23.882840 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:23.884667 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:23.885413 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:23.885622 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:23.886649 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:23.886657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:23.886671 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:23.887170 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:23.887239 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:23.887291 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:23.887387 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:23.887594 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 01:17:23.887832 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":423,"threads_started":1}
I20260812 01:17:23.892261 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:23.897177 27393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab3712acacd4167a10203843d852e0e" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41721 } }
I20260812 01:17:23.897636 27393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:23.897802 27393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab3712acacd4167a10203843d852e0e, State: Initialized, Role: FOLLOWER
I20260812 01:17:23.898461 27393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab3712acacd4167a10203843d852e0e" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41721 } }
I20260812 01:17:23.898756 27393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:23.898911 27393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:23.899114 27393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:23.902113 27393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab3712acacd4167a10203843d852e0e" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41721 } }
I20260812 01:17:23.902762 27393 leader_election.cc:302] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ab3712acacd4167a10203843d852e0e; no voters:
I20260812 01:17:23.903427 27393 leader_election.cc:288] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:23.903569 27414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:23.904271 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:23.897537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48959 (local address 127.20.151.62:41721)
0812 01:17:23.897738 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:23.897744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:23.898161 (+ 417us) server_negotiation.cc:407] Connection header received
0812 01:17:23.898319 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:23.898322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:23.898382 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:23.898459 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:23.899069 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:23.899871 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:23.901669 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:23.902087 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:23.902177 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:23.903361 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:23.903367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:23.903375 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:23.903396 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:23.903449 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:23.903472 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:23.903559 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:23.903806 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 01:17:23.903910 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 01:17:23.905058 27414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [term 1 LEADER]: Becoming Leader. State: Replica: 6ab3712acacd4167a10203843d852e0e, State: Running, Role: LEADER
I20260812 01:17:23.905769 27393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:23.905721 27414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab3712acacd4167a10203843d852e0e" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41721 } }
I20260812 01:17:23.908533 27418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab3712acacd4167a10203843d852e0e. Latest consensus state: current_term: 1 leader_uuid: "6ab3712acacd4167a10203843d852e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab3712acacd4167a10203843d852e0e" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41721 } } }
I20260812 01:17:23.908914 27418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:23.909533 27417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ab3712acacd4167a10203843d852e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab3712acacd4167a10203843d852e0e" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 41721 } } }
I20260812 01:17:23.909871 27417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:23.911913 27433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:23.916317 27433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:23.924580 27433 catalog_manager.cc:1347] Generated new cluster ID: 23e9251fcf9f42918d1b4a1727431955
I20260812 01:17:23.924703 27433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:23.956264 27433 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:23.957634 27433 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:23.967847 27433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ab3712acacd4167a10203843d852e0e: Generated new TSK 0
I20260812 01:17:23.968899 27433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:23.983219 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27292
W20260812 01:17:23.984519 27386 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38115
W20260812 01:17:23.985193 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27292: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:17:23.985343 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 27292
2026-08-12T01:17:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2260 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:24 dist-test-slave-21b9 krb5kdc[27453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:24 dist-test-slave-21b9 krb5kdc[27453](info): set up 2 sockets
Aug 12 01:17:24 dist-test-slave-21b9 krb5kdc[27453](info): commencing operation
krb5kdc: starting...
W20260812 01:17:25.283706 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.003s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:25 dist-test-slave-21b9 krb5kdc[27453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:17:25.670115 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:37317
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:46743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:37317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:26.026168 27477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:26.026523 27477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:26.026697 27477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:26.034353 27477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:26.034490 27477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:26.034550 27477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:26.034669 27477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:26.034734 27477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:26.045477 27477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:46743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:37317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:37317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:26.046872 27477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:26.048292 27477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:26.060418 27488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:26.060388 27487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:26.060444 27490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:26.061751 27477 server_base.cc:1034] running on GCE node
I20260812 01:17:26.062562 27477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:26.064069 27477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:26.065292 27477 hybrid_clock.cc:648] HybridClock initialized: now 1786497446065255 us; error 29 us; skew 500 ppm
I20260812 01:17:26.065626 27477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:26 dist-test-slave-21b9 krb5kdc[27453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497446, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:26.068769 27477 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:17:26.070302 27477 webserver.cc:460] Webserver started at http://127.20.151.62:35501/ using document root <none> and password file <none>
I20260812 01:17:26.070981 27477 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:26.071084 27477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:26.071408 27477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:26.074527 27477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "6e59116ef79e44aea478deef416efd90"
format_stamp: "Formatted at 2026-08-12 01:17:26 on dist-test-slave-21b9"
server_key: "c7509ea2e8b1c6994a9122030df5c98a"
server_key_iv: "97f91d9ddebd4b1241448d2dc2d5abc4"
server_key_version: "encryptionkey@0"
I20260812 01:17:26.075295 27477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "6e59116ef79e44aea478deef416efd90"
format_stamp: "Formatted at 2026-08-12 01:17:26 on dist-test-slave-21b9"
server_key: "c7509ea2e8b1c6994a9122030df5c98a"
server_key_iv: "97f91d9ddebd4b1241448d2dc2d5abc4"
server_key_version: "encryptionkey@0"
I20260812 01:17:26.080598 27477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260812 01:17:26.084231 27503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:26.085734 27477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:17:26.085897 27477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "6e59116ef79e44aea478deef416efd90"
format_stamp: "Formatted at 2026-08-12 01:17:26 on dist-test-slave-21b9"
server_key: "c7509ea2e8b1c6994a9122030df5c98a"
server_key_iv: "97f91d9ddebd4b1241448d2dc2d5abc4"
server_key_version: "encryptionkey@0"
I20260812 01:17:26.086103 27477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:26.092252 27477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:26.096683 27477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:26.096967 27477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:26.115204 27477 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:37317
I20260812 01:17:26.115222 27593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:37317 every 8 connection(s)
I20260812 01:17:26.116739 27477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:26.120596 27594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:26.126613 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 27477
I20260812 01:17:26.126768 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:26.127187 21084 external_mini_cluster.cc:1428] Setting key ed7ab488c29becb360bb082927dfe3a0
I20260812 01:17:26.131639 27594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90: Bootstrap starting.
I20260812 01:17:26.134930 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:26.128971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46449 (local address 127.20.151.62:37317)
0812 01:17:26.129872 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:26.129883 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:26.129927 (+ 44us) server_negotiation.cc:407] Connection header received
0812 01:17:26.131681 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:26.131705 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:26.132068 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:26.132398 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:26.133495 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:26.133502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:26.133514 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:26.133528 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:26.133591 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:26.133611 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:26.133780 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:26.133872 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 01:17:26.134031 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":391,"threads_started":1}
I20260812 01:17:26.135169 27594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:26.136525 27594 log.cc:824] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:26.138880 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:26.139783 27594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90: No bootstrap required, opened a new log
I20260812 01:17:26.146425 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:26.144060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40095 (local address 127.20.151.62:37317)
0812 01:17:26.144267 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:26.144272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:26.144627 (+ 355us) server_negotiation.cc:407] Connection header received
0812 01:17:26.144810 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:26.144814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:26.144865 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:26.144951 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:26.145826 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:26.145830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:26.145833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:17:26.145851 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:26.145890 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:17:26.145908 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:26.146003 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:26.146109 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 01:17:26.146160 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 01:17:26.149508 27594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e59116ef79e44aea478deef416efd90" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37317 } }
I20260812 01:17:26.149904 27594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:17:26.150007 27594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e59116ef79e44aea478deef416efd90, State: Initialized, Role: FOLLOWER
I20260812 01:17:26.150671 27594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e59116ef79e44aea478deef416efd90" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37317 } }
I20260812 01:17:26.150951 27594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:17:26.151100 27594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:17:26.151262 27594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:26.153559 27594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e59116ef79e44aea478deef416efd90" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37317 } }
I20260812 01:17:26.154080 27594 leader_election.cc:302] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e59116ef79e44aea478deef416efd90; no voters:
I20260812 01:17:26.154706 27594 leader_election.cc:288] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:17:26.154807 27610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:26.155500 27610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [term 1 LEADER]: Becoming Leader. State: Replica: 6e59116ef79e44aea478deef416efd90, State: Running, Role: LEADER
I20260812 01:17:26.156042 27610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e59116ef79e44aea478deef416efd90" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37317 } }
I20260812 01:17:26.156821 27594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:26.159476 27612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e59116ef79e44aea478deef416efd90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e59116ef79e44aea478deef416efd90" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37317 } } }
I20260812 01:17:26.158710 27614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e59116ef79e44aea478deef416efd90. Latest consensus state: current_term: 1 leader_uuid: "6e59116ef79e44aea478deef416efd90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e59116ef79e44aea478deef416efd90" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37317 } } }
I20260812 01:17:26.159951 27612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:26.160027 27614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:26.164812 27629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:26.170517 27629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:26.180089 27629 catalog_manager.cc:1347] Generated new cluster ID: 6fd460741ef54e9d91326c37c153d509
I20260812 01:17:26.180246 27629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:26.216773 27629 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:26.217363 27629 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:26.228631 27629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e59116ef79e44aea478deef416efd90: Generated new TSK 0
I20260812 01:17:26.229835 27629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:26.239975 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27477
W20260812 01:17:26.241173 27583 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46449
W20260812 01:17:26.241927 21084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27477: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:17:26.242066 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 27477
2026-08-12T01:17:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2258 ms)
[----------] 18 tests from AuthTokenIssuingTest (41232 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:26 dist-test-slave-21b9 krb5kdc[27657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:26 dist-test-slave-21b9 krb5kdc[27657](info): set up 2 sockets
Aug 12 01:17:26 dist-test-slave-21b9 krb5kdc[27657](info): commencing operation
krb5kdc: starting...
W20260812 01:17:27.545275 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.003s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:27 dist-test-slave-21b9 krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:17:27.928229 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:44155
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:44155,127.20.151.61:33289,127.20.151.60:35047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:28.285753 27682 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:17:28.286135 27682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:28.286247 27682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:28.286367 27682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:28.293711 27682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:28.293818 27682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:28.293905 27682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:28.293975 27682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:28.294077 27682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:28.304162 27682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:44155,127.20.151.61:33289,127.20.151.60:35047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:44155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:28.305517 27682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:28.306859 27682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:28.318763 27690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:28.318810 27691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:28.318810 27693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:28.320135 27682 server_base.cc:1034] running on GCE node
I20260812 01:17:28.321003 27682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:28.322450 27682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:28.323693 27682 hybrid_clock.cc:648] HybridClock initialized: now 1786497448323628 us; error 27 us; skew 500 ppm
I20260812 01:17:28.324041 27682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:28 dist-test-slave-21b9 krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497448, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:28.327371 27682 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:17:28.328949 27682 webserver.cc:460] Webserver started at http://127.20.151.62:41861/ using document root <none> and password file <none>
I20260812 01:17:28.329675 27682 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:28.329782 27682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:28.330109 27682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:28.333190 27682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "be16ab17aacf46da859dd75c72cc7313"
format_stamp: "Formatted at 2026-08-12 01:17:28 on dist-test-slave-21b9"
server_key: "c21ad5aa2483276071d9c045e1683932"
server_key_iv: "bb29c1af9b3be3e5bce7d4d608ad4f5a"
server_key_version: "encryptionkey@0"
I20260812 01:17:28.334010 27682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "be16ab17aacf46da859dd75c72cc7313"
format_stamp: "Formatted at 2026-08-12 01:17:28 on dist-test-slave-21b9"
server_key: "c21ad5aa2483276071d9c045e1683932"
server_key_iv: "bb29c1af9b3be3e5bce7d4d608ad4f5a"
server_key_version: "encryptionkey@0"
I20260812 01:17:28.339257 27682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:17:28.342938 27701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:28.344452 27682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:17:28.344650 27682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "be16ab17aacf46da859dd75c72cc7313"
format_stamp: "Formatted at 2026-08-12 01:17:28 on dist-test-slave-21b9"
server_key: "c21ad5aa2483276071d9c045e1683932"
server_key_iv: "bb29c1af9b3be3e5bce7d4d608ad4f5a"
server_key_version: "encryptionkey@0"
I20260812 01:17:28.344882 27682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:28.376291 27682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:28.380466 27682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:28.380750 27682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:28.399771 27682 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:44155
I20260812 01:17:28.399808 27786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:44155 every 8 connection(s)
I20260812 01:17:28.401108 27682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:28.405123 27787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:28.406358 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 27682
I20260812 01:17:28.406654 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:28.407639 21084 external_mini_cluster.cc:1428] Setting key e830ff800ea90d4a5bf3ea6fcb421318
I20260812 01:17:28.414480 27787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:17:28 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497448, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:17:28.437985 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.418290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:44155 (local address 127.0.0.1:38542)
0812 01:17:28.419177 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.419223 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:28.421253 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:28.422049 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:28.422056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:28.422762 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:28.423110 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:28.423129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:28.424683 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:28.424688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:28.426012 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:28.426027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:28.427082 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:28.427089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:28.427301 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:28.427957 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:28.427985 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:28.430579 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:28.433731 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:28.433743 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:28.433756 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:28.434151 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:28.434545 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:28.434550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:28.434554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:28.434721 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:28.435537 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:28.435546 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:28.435763 (+ 217us) client_negotiation.cc:769] Sending connection context
0812 01:17:28.435992 (+ 229us) client_negotiation.cc:240] Negotiation successful
0812 01:17:28.436627 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":125,"thread_start_us":370,"threads_started":1}
I20260812 01:17:28.438030 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.418291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.20.151.62:44155)
0812 01:17:28.419993 (+ 1702us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:28.420005 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:28.420059 (+ 54us) server_negotiation.cc:407] Connection header received
0812 01:17:28.421613 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:28.421618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:28.421839 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:28.422009 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:28.423342 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:28.424540 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:28.426228 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:28.426814 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:28.426961 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:28.430806 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:28.430815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:28.430827 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:28.430857 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:28.433597 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:28.434276 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:28.434283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:28.434285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:28.434433 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:28.434824 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:28.434827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:28.434829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:28.435408 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:28.435605 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:28.436137 (+ 532us) server_negotiation.cc:299] Negotiation successful
0812 01:17:28.436437 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1497,"thread_start_us":1269,"threads_started":1}
I20260812 01:17:28.441293 27787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:17:28.442432 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.442013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:47932)
0812 01:17:28.442198 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.442312 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 01:17:28.442739 27702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.20.151.61@KRBTEST.COM; defaulting to no policy
I20260812 01:17:28.444339 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.443996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:47936)
0812 01:17:28.444140 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.444208 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 01:17:28.444522 27787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:33289: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab.
Entry for principal kudu/127.20.151.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab.
I20260812 01:17:28.488437 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.61:33289
--webserver_interface=127.20.151.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab
--principal=kudu/127.20.151.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:44155,127.20.151.61:33289,127.20.151.60:35047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:17:28.493872 27787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } attempt: 1
I20260812 01:17:28.495180 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.494649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:47948)
0812 01:17:28.494866 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.495008 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 01:17:28.496248 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.495866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:47954)
0812 01:17:28.496011 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.496092 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 01:17:28.496456 27787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:33289: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
I20260812 01:17:28.564728 27787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } attempt: 2
I20260812 01:17:28.566058 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.565557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:47966)
0812 01:17:28.565803 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.565900 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 01:17:28.567099 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.566669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:47972)
0812 01:17:28.566832 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.566937 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 01:17:28.567319 27787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:33289: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
I20260812 01:17:28.658569 27787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } attempt: 3
I20260812 01:17:28.659878 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.659366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:47982)
0812 01:17:28.659630 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.659729 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260812 01:17:28.660835 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.660499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:47998)
0812 01:17:28.660636 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.660705 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 01:17:28.661021 27787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:33289: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
I20260812 01:17:28.804294 27787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } attempt: 4
I20260812 01:17:28.805663 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.805062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:48010)
0812 01:17:28.805424 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.805518 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260812 01:17:28.806660 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.806308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:48012)
0812 01:17:28.806453 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.806535 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 01:17:28.806860 27787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:33289: Network error: Client connection negotiation failed: client connection to 127.20.151.61:33289: connect: Connection refused (error 111)
W20260812 01:17:28.855547 27796 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:17:28.855983 27796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:28.856101 27796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:28.856228 27796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:28.863911 27796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:28.864030 27796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:28.864136 27796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:28.864214 27796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:28.864310 27796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:28.874805 27796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:44155,127.20.151.61:33289,127.20.151.60:35047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab
--principal=kudu/127.20.151.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.61:33289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.151.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:28.876250 27796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:28.877655 27796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:28.889549 27809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:28.889624 27803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:28.889660 27801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:28.890513 27796 server_base.cc:1034] running on GCE node
I20260812 01:17:28.891342 27796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:28.892788 27796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:28.894037 27796 hybrid_clock.cc:648] HybridClock initialized: now 1786497448893956 us; error 49 us; skew 500 ppm
I20260812 01:17:28.894379 27796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:28 dist-test-slave-21b9 krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497448, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:28.897557 27796 init.cc:377] Logged in from keytab as kudu/127.20.151.61@KRBTEST.COM (short username kudu)
I20260812 01:17:28.898976 27796 webserver.cc:460] Webserver started at http://127.20.151.61:37143/ using document root <none> and password file <none>
I20260812 01:17:28.899718 27796 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:28.899827 27796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:28.900137 27796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:28.903113 27796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/data/instance:
uuid: "70d7787b689c4487a442a03c3384778d"
format_stamp: "Formatted at 2026-08-12 01:17:28 on dist-test-slave-21b9"
server_key: "a3eea9e6b054d0101414f10d2fa71b5c"
server_key_iv: "8cd40dfb3dd163e99380fa8d39486447"
server_key_version: "encryptionkey@0"
I20260812 01:17:28.903972 27796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/wal/instance:
uuid: "70d7787b689c4487a442a03c3384778d"
format_stamp: "Formatted at 2026-08-12 01:17:28 on dist-test-slave-21b9"
server_key: "a3eea9e6b054d0101414f10d2fa71b5c"
server_key_iv: "8cd40dfb3dd163e99380fa8d39486447"
server_key_version: "encryptionkey@0"
I20260812 01:17:28.909227 27796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:17:28.912792 27828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:28.914165 27796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:17:28.914341 27796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/wal
uuid: "70d7787b689c4487a442a03c3384778d"
format_stamp: "Formatted at 2026-08-12 01:17:28 on dist-test-slave-21b9"
server_key: "a3eea9e6b054d0101414f10d2fa71b5c"
server_key_iv: "8cd40dfb3dd163e99380fa8d39486447"
server_key_version: "encryptionkey@0"
I20260812 01:17:28.914573 27796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:28.936818 27796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:28.941327 27796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:28.941610 27796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:28.960464 27796 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.61:33289
I20260812 01:17:28.960485 27910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.61:33289 every 8 connection(s)
I20260812 01:17:28.961884 27796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
I20260812 01:17:28.965845 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:28.965848 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 27796
I20260812 01:17:28.966030 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-1/wal/instance
I20260812 01:17:28.966566 21084 external_mini_cluster.cc:1428] Setting key 89c483cc9a7efa3a3e3edb27058d3176
I20260812 01:17:28.974792 27911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:17:28 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497448, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:17:28.997653 27916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.978106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.20.151.62:44155)
0812 01:17:28.978700 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:28.978706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:28.979778 (+ 1072us) server_negotiation.cc:407] Connection header received
0812 01:17:28.981765 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:28.981771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:28.981855 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:28.981972 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:28.983622 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:28.984551 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:28.986839 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:28.987310 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:28.987414 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:28.991613 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:28.991618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:28.991622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:28.991661 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:28.994248 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:28.995145 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:28.995150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:28.995152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:28.995306 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:28.995848 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:28.995852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:28.995854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:28.996169 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:28.996283 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:28.997169 (+ 886us) server_negotiation.cc:299] Negotiation successful
0812 01:17:28.997346 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":311,"threads_started":1}
I20260812 01:17:28.999008 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:28.978725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:44155 (local address 127.0.0.1:38554)
0812 01:17:28.979597 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:28.979645 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:28.981481 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:28.982191 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:28.982200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:28.982760 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:28.983400 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:28.983430 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:28.984740 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:28.984746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:28.986692 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:28.986708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:28.987569 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:28.987576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:28.987847 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:28.988618 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:28.988653 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:28.991350 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:28.994394 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:28.994403 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:28.994420 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:28.994995 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:28.995411 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:28.995415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:28.995418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:28.995721 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:28.996292 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:28.996299 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:28.996764 (+ 465us) client_negotiation.cc:769] Sending connection context
0812 01:17:28.997109 (+ 345us) client_negotiation.cc:240] Negotiation successful
0812 01:17:28.997647 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":353,"threads_started":1}
I20260812 01:17:29.001042 27911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.151.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.60@KRBTEST.COM" created.
Aug 12 01:17:29 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497448, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
I20260812 01:17:29.017084 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.001870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:48028)
0812 01:17:29.002212 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.002232 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:29.002349 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:29.003530 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:29.003536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:29.003927 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:29.004084 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.004093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.006168 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.006173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.007120 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.007133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.008129 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.008134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.008290 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.008943 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:29.008963 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:29.011374 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:29.014596 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.014602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.014606 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.015004 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.015384 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.015389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.015393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.015527 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.016134 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:29.016138 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:29.016277 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 01:17:29.016394 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 01:17:29.016685 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260812 01:17:29.017204 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.002132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.20.151.61:33289)
0812 01:17:29.002917 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.002926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.002963 (+ 37us) server_negotiation.cc:407] Connection header received
0812 01:17:29.003023 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.003033 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.003335 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.003505 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.004280 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.005999 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.007301 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.007956 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.008058 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.011588 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.011602 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.011619 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.011654 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.014469 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.015126 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.015134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.015136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.015255 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.015633 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.015638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.015642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.015998 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.016193 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.016534 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.016805 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":581,"thread_start_us":418,"threads_started":1}
I20260812 01:17:29.020136 27911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:17:29.021313 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.020844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44092)
0812 01:17:29.021028 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.021149 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 01:17:29.021552 27829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
I20260812 01:17:29.023073 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.022755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44094)
0812 01:17:29.022891 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.022958 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 01:17:29.023307 27911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:35047: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab.
Entry for principal kudu/127.20.151.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab.
I20260812 01:17:29.054888 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.60:35047
--webserver_interface=127.20.151.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab
--principal=kudu/127.20.151.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:33847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:44155,127.20.151.61:33289,127.20.151.60:35047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:17:29.072646 27911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } attempt: 1
I20260812 01:17:29.073997 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.073412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44106)
0812 01:17:29.073663 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.073801 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 01:17:29.075059 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.074673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44122)
0812 01:17:29.074834 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.074926 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 01:17:29.075301 27911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:35047: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
I20260812 01:17:29.106133 27787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } attempt: 5
Aug 12 01:17:29 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497448, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
I20260812 01:17:29.121881 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.107097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.20.151.61:33289)
0812 01:17:29.107342 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.107349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.107371 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:17:29.107470 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.107507 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.107577 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.107682 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.108799 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.109524 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.111242 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.111939 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.112051 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.116095 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.116101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.116104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.116137 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.118871 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.119801 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.119806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.119808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.119942 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.120430 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.120435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.120437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.120757 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.120878 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.121442 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.121652 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 01:17:29.122051 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.106965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:48038)
0812 01:17:29.107164 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.107188 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:29.107342 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:29.107777 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:29.107783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:29.108341 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:29.108622 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.108637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.109669 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.109676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.111077 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.111096 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.112110 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.112117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.112357 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.113066 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:29.113094 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:29.115856 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:29.119012 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.119018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.119022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.119647 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.120069 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.120074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.120078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.120272 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.120898 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:29.120905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:29.121100 (+ 195us) client_negotiation.cc:769] Sending connection context
0812 01:17:29.121280 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 01:17:29.121628 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 01:17:29.123366 27787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:17:29.124506 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.124030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44128)
0812 01:17:29.124216 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.124320 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 01:17:29.125593 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.125178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44138)
0812 01:17:29.125338 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.125435 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 01:17:29.125850 27787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:35047: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
I20260812 01:17:29.143533 27911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } attempt: 2
I20260812 01:17:29.144917 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.144313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44152)
0812 01:17:29.144554 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.144684 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 01:17:29.146154 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.145704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44166)
0812 01:17:29.145872 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.145988 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 01:17:29.146422 27911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:35047: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
I20260812 01:17:29.177165 27787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } attempt: 1
I20260812 01:17:29.178592 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.177967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44180)
0812 01:17:29.178221 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.178350 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 01:17:29.179812 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.179381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44184)
0812 01:17:29.179547 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.179655 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 01:17:29.180109 27787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:35047: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
I20260812 01:17:29.237735 27911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } attempt: 3
I20260812 01:17:29.239180 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.238554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44194)
0812 01:17:29.238818 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.238951 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 01:17:29.240464 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.239920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44204)
0812 01:17:29.240190 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.240297 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260812 01:17:29.240693 27911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:35047: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
I20260812 01:17:29.248370 27787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } attempt: 2
I20260812 01:17:29.249662 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.249073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44220)
0812 01:17:29.249350 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.249459 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260812 01:17:29.250710 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.250365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44236)
0812 01:17:29.250516 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.250598 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 01:17:29.250903 27787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:35047: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
I20260812 01:17:29.357133 27787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } attempt: 3
I20260812 01:17:29.358320 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.357857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44250)
0812 01:17:29.358053 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.358148 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 01:17:29.359429 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.359016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44262)
0812 01:17:29.359204 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.359314 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 01:17:29.359637 27787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:35047: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
I20260812 01:17:29.383911 27911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } attempt: 4
I20260812 01:17:29.385174 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.384609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44278)
0812 01:17:29.384857 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.384982 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 01:17:29.386261 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.385870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44282)
0812 01:17:29.386029 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.386136 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 01:17:29.386492 27911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:35047: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
W20260812 01:17:29.426677 27923 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:17:29.427028 27923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:29.427129 27923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:29.427243 27923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:29.434731 27923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:29.434829 27923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:29.434914 27923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:29.434978 27923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:29.435065 27923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:29.445256 27923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:33847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:44155,127.20.151.61:33289,127.20.151.60:35047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab
--principal=kudu/127.20.151.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.60:35047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.151.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:29.446643 27923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:29.448096 27923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:29.460081 27930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:29.460119 27931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:29.460145 27934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:29.460968 27923 server_base.cc:1034] running on GCE node
I20260812 01:17:29.461763 27923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:29.463176 27923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:29.464408 27923 hybrid_clock.cc:648] HybridClock initialized: now 1786497449464337 us; error 39 us; skew 500 ppm
I20260812 01:17:29.464751 27923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:29 dist-test-slave-21b9 krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497449, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:29.468006 27923 init.cc:377] Logged in from keytab as kudu/127.20.151.60@KRBTEST.COM (short username kudu)
I20260812 01:17:29.469513 27923 webserver.cc:460] Webserver started at http://127.20.151.60:38043/ using document root <none> and password file <none>
I20260812 01:17:29.470207 27923 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:29.470299 27923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:29.470589 27923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:29.473642 27923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/data/instance:
uuid: "bfa288741376486489f54a2a0d2ef2b7"
format_stamp: "Formatted at 2026-08-12 01:17:29 on dist-test-slave-21b9"
server_key: "7461a38ecaec77ed70a88de99a59da46"
server_key_iv: "eb568757eaface21ff5f8177dfbe12d3"
server_key_version: "encryptionkey@0"
I20260812 01:17:29.474436 27923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/wal/instance:
uuid: "bfa288741376486489f54a2a0d2ef2b7"
format_stamp: "Formatted at 2026-08-12 01:17:29 on dist-test-slave-21b9"
server_key: "7461a38ecaec77ed70a88de99a59da46"
server_key_iv: "eb568757eaface21ff5f8177dfbe12d3"
server_key_version: "encryptionkey@0"
I20260812 01:17:29.479779 27923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 01:17:29.483371 27950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:29.484757 27923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:17:29.484918 27923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/wal
uuid: "bfa288741376486489f54a2a0d2ef2b7"
format_stamp: "Formatted at 2026-08-12 01:17:29 on dist-test-slave-21b9"
server_key: "7461a38ecaec77ed70a88de99a59da46"
server_key_iv: "eb568757eaface21ff5f8177dfbe12d3"
server_key_version: "encryptionkey@0"
I20260812 01:17:29.485159 27923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:29.524533 27923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:29.529029 27923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:29.529352 27923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:29.536844 27787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } attempt: 4
I20260812 01:17:29.538084 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.537599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44290)
0812 01:17:29.537799 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.537909 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 01:17:29.539377 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.538857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44304)
0812 01:17:29.539033 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.539163 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 01:17:29.539664 27787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:35047: Network error: Client connection negotiation failed: client connection to 127.20.151.60:35047: connect: Connection refused (error 111)
I20260812 01:17:29.548185 27923 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.60:35047
I20260812 01:17:29.548205 28042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.60:35047 every 8 connection(s)
I20260812 01:17:29.549695 27923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
I20260812 01:17:29.553627 28043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:29.556135 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 27923
I20260812 01:17:29.556347 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786497066589499-21084-0/minicluster-data/master-2/wal/instance
I20260812 01:17:29.556872 21084 external_mini_cluster.cc:1428] Setting key 5e4b89a4e0c65dc75a82a7c3b073f06c
I20260812 01:17:29.561731 28043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:17:29 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
Aug 12 01:17:29 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497449, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:17:29.580312 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.564682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38556 (local address 127.20.151.62:44155)
0812 01:17:29.565205 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.565211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.565657 (+ 446us) server_negotiation.cc:407] Connection header received
0812 01:17:29.565826 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.565830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.565894 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.565989 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.567463 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.568466 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.570358 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.570811 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.570913 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.574500 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.574513 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.574517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.574557 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.577239 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.578055 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.578059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.578062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.578206 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.578631 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.578635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.578637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.578919 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.579065 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.579755 (+ 690us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.579983 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":217,"threads_started":1}
I20260812 01:17:29.581490 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.564692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:44155 (local address 127.0.0.1:38556)
0812 01:17:29.565516 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.565562 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:29.565708 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:29.566079 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:29.566086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:29.566946 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:29.567308 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.567321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.568675 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.568680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.570173 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.570190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.571061 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.571069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.571261 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.571930 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:29.571964 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:29.574310 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:29.577353 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.577363 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.577370 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.577930 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.578318 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.578324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.578327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.578519 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.579054 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:29.579063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:29.579337 (+ 274us) client_negotiation.cc:769] Sending connection context
0812 01:17:29.579622 (+ 285us) client_negotiation.cc:240] Negotiation successful
0812 01:17:29.580170 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":171,"spinlock_wait_cycles":27392,"thread_start_us":452,"threads_started":1}
I20260812 01:17:29.581509 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.558890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.20.151.60:35047)
0812 01:17:29.559746 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.559758 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.559807 (+ 49us) server_negotiation.cc:407] Connection header received
0812 01:17:29.561537 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.561561 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.561954 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.562294 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.564145 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.566084 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.567765 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.568750 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.568892 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.573619 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.573633 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.573650 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.573682 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.576616 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.577339 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.577348 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.577357 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.577580 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.578004 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.578008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.578012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.578789 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.579054 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.579554 (+ 500us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.580031 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":639,"thread_start_us":409,"threads_started":1}
I20260812 01:17:29.583458 28043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:17:29 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497449, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
Aug 12 01:17:29 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:17:29.598685 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.584346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:33289 (local address 127.0.0.1:48046)
0812 01:17:29.584912 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.584930 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:29.585038 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:29.585381 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:29.585385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:29.585832 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:29.586076 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.586090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.587237 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.587242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.588557 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.588574 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.589356 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.589360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.589560 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.590170 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:29.590187 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:29.592542 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:29.596025 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.596031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.596036 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.596449 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.596907 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.596911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.596913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.597062 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.597697 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:29.597703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:29.597891 (+ 188us) client_negotiation.cc:769] Sending connection context
0812 01:17:29.598096 (+ 205us) client_negotiation.cc:240] Negotiation successful
0812 01:17:29.598364 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458}
I20260812 01:17:29.598980 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.584489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.20.151.61:33289)
0812 01:17:29.584713 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.584719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.584989 (+ 270us) server_negotiation.cc:407] Connection header received
0812 01:17:29.585161 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.585166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.585243 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.585335 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.586215 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.587076 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.588701 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.589210 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.589322 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.592737 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.592744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.592749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.592791 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.595884 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.596608 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.596614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.596617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.596784 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.597173 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.597178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.597181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.597558 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.597672 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.598244 (+ 572us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.598487 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 01:17:29.600034 28043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:17:29.600368 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.584721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.20.151.62:44155)
0812 01:17:29.584862 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.584865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.584896 (+ 31us) server_negotiation.cc:407] Connection header received
0812 01:17:29.585093 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.585098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.585185 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.585282 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.586694 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.587602 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.589166 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.589756 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.589864 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.594257 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.594263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.594267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.594302 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.597380 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.598231 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.598237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.598240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.598404 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.598904 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.598909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.598912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.599253 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.599573 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.599885 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.600103 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Aug 12 01:17:29 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497449, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
Aug 12 01:17:29 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
I20260812 01:17:29.613411 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.600792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44324)
0812 01:17:29.600956 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.600977 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:29.601054 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:29.601405 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:29.601408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:29.601802 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:29.601945 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.601953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.603114 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.603118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.603991 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.604003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.604661 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.604666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.604884 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.605513 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:29.605535 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:29.607721 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:29.610800 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.610806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.610810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.611345 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.611775 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.611778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.611781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.611915 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.612535 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:29.612539 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:29.612687 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 01:17:29.612802 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 01:17:29.613108 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":20}
I20260812 01:17:29.613840 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.600986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.20.151.60:35047)
0812 01:17:29.601148 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.601154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.601172 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:17:29.601221 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.601226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.601297 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.601401 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.602073 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.602996 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.604122 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.604528 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.604651 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.607922 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.607927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.607931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.607960 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.610660 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.611449 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.611454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.611457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.611657 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.612032 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.612036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.612038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.612386 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.612550 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.612949 (+ 399us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.613291 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":50}
I20260812 01:17:29.617573 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.602267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.20.151.61:33289)
0812 01:17:29.602429 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.602434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.602465 (+ 31us) server_negotiation.cc:407] Connection header received
0812 01:17:29.602513 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.602518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.602585 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.602680 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.603894 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.604625 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.606078 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.606540 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.606653 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.611368 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.611373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.611377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.611418 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.614643 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.615492 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.615499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.615502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.615706 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.616115 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.616120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.616123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.616470 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.616737 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.616961 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.617206 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:17:29.620195 28043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7: Bootstrap starting.
I20260812 01:17:29.622769 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:29.624197 28043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:29.625614 28043 log.cc:824] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:29.628435 28043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7: No bootstrap required, opened a new log
I20260812 01:17:29.626305 27743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "be16ab17aacf46da859dd75c72cc7313"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38570
I20260812 01:17:29.635942 28043 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } }
I20260812 01:17:29.636307 28043 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa288741376486489f54a2a0d2ef2b7, State: Initialized, Role: FOLLOWER
I20260812 01:17:29.636767 28043 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } }
I20260812 01:17:29.637651 28057 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } } }
I20260812 01:17:29.637954 28057 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:29.638971 28043 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 01:17:29.648458 28074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:17:29.648609 28074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:17:29.646457 27923 master_runner.cc:186] Error getting master registration for 127.20.151.62:44155: OK, instance_id { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" instance_seqno: 1786497448394356 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:17:29.657691 27923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 01:17:29.685768 27911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } attempt: 5
Aug 12 01:17:29 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497448, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
I20260812 01:17:29.702288 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.686563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44326)
0812 01:17:29.686790 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.686821 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:29.686970 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:29.687350 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:29.687356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:29.687795 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:29.687969 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.687989 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.689401 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.689407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.690329 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.690346 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.691403 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.691411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.691640 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.692357 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:29.692383 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:29.695187 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:29.699248 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.699254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.699259 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.699859 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.700267 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.700273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.700276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.700458 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.701206 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:29.701211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:29.701392 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 01:17:29.701548 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 01:17:29.701878 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 01:17:29.702297 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.686666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.20.151.60:35047)
0812 01:17:29.686863 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.686869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.686892 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:17:29.687122 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.687128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.687203 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.687337 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.688142 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.689251 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.690483 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.691227 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.691369 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.695411 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.695416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.695419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.695453 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.699075 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.699983 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.699989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.699993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.700150 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.700576 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.700581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.700584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.701078 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.701226 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.701674 (+ 448us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.701914 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 01:17:29.706945 27911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d: Bootstrap starting.
I20260812 01:17:29.710259 27911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:29.711370 27911 log.cc:824] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:29.714223 27911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d: No bootstrap required, opened a new log
I20260812 01:17:29.722059 27911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } }
I20260812 01:17:29.722422 27911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d7787b689c4487a442a03c3384778d, State: Initialized, Role: FOLLOWER
I20260812 01:17:29.722932 27911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } }
I20260812 01:17:29.723929 28077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } } }
I20260812 01:17:29.724344 28077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:29.725137 27911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [sys.catalog]: configured and running, proceeding with master startup.
W20260812 01:17:29.735581 28089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:17:29.735786 28089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:17:29.732636 27796 master_runner.cc:186] Error getting master registration for 127.20.151.62:44155: OK, instance_id { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" instance_seqno: 1786497448394356 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:17:29.741746 27796 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 01:17:29.781860 27961 tablet.cc:2369] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:17:29.816928 27787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } attempt: 5
Aug 12 01:17:29 dist-test-slave-21b9 krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497448, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
I20260812 01:17:29.832667 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.817837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.20.151.60:35047)
0812 01:17:29.818035 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.818041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.818188 (+ 147us) server_negotiation.cc:407] Connection header received
0812 01:17:29.818413 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.818419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.818534 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.818629 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.819639 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.820786 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.822069 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.822757 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.822875 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.826496 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.826501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.826505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.826537 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.829662 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.830536 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.830542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.830546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.830693 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.831141 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.831146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.831150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.831519 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.831664 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.832072 (+ 408us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.832292 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:17:29.832662 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.817741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:35047 (local address 127.0.0.1:44328)
0812 01:17:29.818089 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:29.818109 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:29.818254 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:29.818677 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:29.818682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:29.819205 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:29.819455 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.819469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.820926 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.820932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.821881 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:29.821911 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.822902 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.822909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:29.823107 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.823746 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:29.823763 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:29.826292 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:29.829907 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.829913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.829917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.830429 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.830803 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:29.830809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:29.830813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:29.831012 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:29.831636 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:29.831641 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:29.831814 (+ 173us) client_negotiation.cc:769] Sending connection context
0812 01:17:29.831943 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 01:17:29.832232 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260812 01:17:29.837740 27787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313: Bootstrap starting.
I20260812 01:17:29.841957 27787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:29.843345 27787 log.cc:824] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:29.847860 27787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313: No bootstrap required, opened a new log
I20260812 01:17:29.856107 27787 raft_consensus.cc:348] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } }
I20260812 01:17:29.856611 27787 raft_consensus.cc:729] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be16ab17aacf46da859dd75c72cc7313, State: Initialized, Role: FOLLOWER
I20260812 01:17:29.857199 27787 consensus_queue.cc:260] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } }
I20260812 01:17:29.858397 28093 sys_catalog.cc:455] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } } }
I20260812 01:17:29.858804 28093 sys_catalog.cc:458] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:29.859860 27787 sys_catalog.cc:564] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:29.860062 27743 raft_consensus.cc:482] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 01:17:29.860374 27743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:29.864238 27743 raft_consensus.cc:504] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } }
I20260812 01:17:29.866460 27743 leader_election.cc:288] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [CANDIDATE]: Term 1 election: Requested vote from peers 70d7787b689c4487a442a03c3384778d (127.20.151.61:33289), bfa288741376486489f54a2a0d2ef2b7 (127.20.151.60:35047)
I20260812 01:17:29.867782 27996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be16ab17aacf46da859dd75c72cc7313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bfa288741376486489f54a2a0d2ef2b7"
I20260812 01:17:29.868287 27996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:29.871346 27868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be16ab17aacf46da859dd75c72cc7313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "70d7787b689c4487a442a03c3384778d"
I20260812 01:17:29.871922 27868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:29.876037 27996 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be16ab17aacf46da859dd75c72cc7313 in term 1.
I20260812 01:17:29.876372 27868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be16ab17aacf46da859dd75c72cc7313 in term 1.
I20260812 01:17:29.877225 27702 leader_election.cc:302] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be16ab17aacf46da859dd75c72cc7313, bfa288741376486489f54a2a0d2ef2b7; no voters:
I20260812 01:17:29.877748 28093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:29.879537 28093 raft_consensus.cc:686] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [term 1 LEADER]: Becoming Leader. State: Replica: be16ab17aacf46da859dd75c72cc7313, State: Running, Role: LEADER
I20260812 01:17:29.880182 28093 consensus_queue.cc:237] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } }
W20260812 01:17:29.883942 27711 tablet.cc:2369] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:17:29.892465 28104 sys_catalog.cc:455] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be16ab17aacf46da859dd75c72cc7313. Latest consensus state: current_term: 1 leader_uuid: "be16ab17aacf46da859dd75c72cc7313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } } }
I20260812 01:17:29.893009 28104 sys_catalog.cc:458] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:29.908443 28112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:29.925295 28112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260812 01:17:29.944450 27841 tablet.cc:2369] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:17:29.950582 27996 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [term 1 FOLLOWER]: Refusing update from remote peer be16ab17aacf46da859dd75c72cc7313: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:17:29.955714 28104 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:17:29.969911 27868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [term 1 FOLLOWER]: Refusing update from remote peer be16ab17aacf46da859dd75c72cc7313: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:17:29.970822 28104 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:17:29.971184 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.909122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.20.151.62:44155)
0812 01:17:29.909337 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.909343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.926871 (+ 17528us) server_negotiation.cc:407] Connection header received
0812 01:17:29.929843 (+ 2972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.929850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.929944 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.930037 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.937225 (+ 7188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.938432 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.947757 (+ 9325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.948456 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.948558 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.959236 (+ 10678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.959242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.959246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.959330 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.963128 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.967517 (+ 4389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.967524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.967527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.967681 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.968490 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.968497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.968500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.968906 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.969213 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.970509 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.970740 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 01:17:29.974444 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.911201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.20.151.61:33289)
0812 01:17:29.911401 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.911407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.933740 (+ 22333us) server_negotiation.cc:407] Connection header received
0812 01:17:29.934194 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.934201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.934291 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.934361 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.942701 (+ 8340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.943866 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.953420 (+ 9554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.954100 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.954197 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.961154 (+ 6957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.961162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.961166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.961209 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.964860 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.969764 (+ 4904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.969771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.969775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.969941 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.971935 (+ 1994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.971942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.971945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.972365 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.972611 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.973806 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.974085 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 01:17:29.977149 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:29.927530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.20.151.60:35047)
0812 01:17:29.927755 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:29.927762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:29.940178 (+ 12416us) server_negotiation.cc:407] Connection header received
0812 01:17:29.940426 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:29.940433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:29.940515 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:29.940587 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:29.952231 (+ 11644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.953224 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.958954 (+ 5730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:29.959732 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:29.959888 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:29.967192 (+ 7304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:29.967199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:29.967203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:29.967245 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:29.970779 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.973646 (+ 2867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.973652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.973655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.973851 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:29.975518 (+ 1667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:29.975524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:29.975527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:29.975883 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:29.976066 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:29.976547 (+ 481us) server_negotiation.cc:299] Negotiation successful
0812 01:17:29.976801 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 01:17:29.977399 28057 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be16ab17aacf46da859dd75c72cc7313. Latest consensus state: current_term: 1 leader_uuid: "be16ab17aacf46da859dd75c72cc7313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } } }
I20260812 01:17:29.977741 28057 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:29.978394 28130 mvcc.cc:205] Tried to move back new op lower bound from 7317493554979778560 to 7317493554715852800. Current Snapshot: MvccSnapshot[applied={T|T < 7317493554979778560}]
I20260812 01:17:29.986277 28057 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be16ab17aacf46da859dd75c72cc7313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } } }
I20260812 01:17:29.986550 28057 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:29.993638 28104 sys_catalog.cc:455] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be16ab17aacf46da859dd75c72cc7313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } } }
I20260812 01:17:29.993940 28104 sys_catalog.cc:458] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:29.995585 28077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [sys.catalog]: SysCatalogTable state changed. Reason: New leader be16ab17aacf46da859dd75c72cc7313. Latest consensus state: current_term: 1 leader_uuid: "be16ab17aacf46da859dd75c72cc7313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } } }
I20260812 01:17:29.995925 28077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:29.997365 28112 catalog_manager.cc:1347] Generated new cluster ID: c3090ad5e8a24ab7a6890bf1c14a00e0
I20260812 01:17:29.997473 28112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:30.021216 28093 sys_catalog.cc:455] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be16ab17aacf46da859dd75c72cc7313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } } }
I20260812 01:17:30.021605 28093 sys_catalog.cc:458] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:30.023662 28135 mvcc.cc:205] Tried to move back new op lower bound from 7317493554979778560 to 7317493554715852800. Current Snapshot: MvccSnapshot[applied={T|T < 7317493554979778560}]
I20260812 01:17:30.029378 28077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be16ab17aacf46da859dd75c72cc7313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be16ab17aacf46da859dd75c72cc7313" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 44155 } } peers { permanent_uuid: "70d7787b689c4487a442a03c3384778d" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 33289 } } peers { permanent_uuid: "bfa288741376486489f54a2a0d2ef2b7" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 35047 } } }
I20260812 01:17:30.029672 28077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:30.063256 28112 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:17:30.065423 28112 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:30.099045 28112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be16ab17aacf46da859dd75c72cc7313: Generated new TSK 0
I20260812 01:17:30.100201 28112 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 01:17:30.257895 28053 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.244239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.20.151.62:44155)
0812 01:17:30.244491 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.244498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.255795 (+ 11297us) server_negotiation.cc:407] Connection header received
0812 01:17:30.256049 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.256055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.256145 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.256251 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.257621 (+ 1370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38588: BlockingRecv error: recv got EOF from 127.0.0.1:38588 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 01:17:30.259452 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.244685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.20.151.61:33289)
0812 01:17:30.244904 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.244910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.255676 (+ 10766us) server_negotiation.cc:407] Connection header received
0812 01:17:30.257780 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.257786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.257882 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.257960 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.259174 (+ 1214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48086: BlockingRecv error: recv got EOF from 127.0.0.1:48086 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 01:17:30.260499 28052 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.256883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.20.151.60:35047)
0812 01:17:30.257059 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.257064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.258683 (+ 1619us) server_negotiation.cc:407] Connection header received
0812 01:17:30.259127 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.259133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.259214 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.259317 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.260257 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44356: BlockingRecv error: recv got EOF from 127.0.0.1:44356 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 01:17:30.267884 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.265900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.20.151.61:33289)
0812 01:17:30.266079 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.266086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.266391 (+ 305us) server_negotiation.cc:407] Connection header received
0812 01:17:30.266594 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.266598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.266694 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.266798 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.267577 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48102: BlockingRecv error: recv got EOF from 127.0.0.1:48102 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260812 01:17:30.279400 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.277252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.20.151.61:33289)
0812 01:17:30.277456 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.277466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.277991 (+ 525us) server_negotiation.cc:407] Connection header received
0812 01:17:30.278180 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.278184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.278263 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.278351 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.279131 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48112: BlockingRecv error: recv got EOF from 127.0.0.1:48112 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260812 01:17:30.286481 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.284388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48124 (local address 127.20.151.61:33289)
0812 01:17:30.284587 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.284594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.285089 (+ 495us) server_negotiation.cc:407] Connection header received
0812 01:17:30.285313 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.285317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.285406 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.285500 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.286219 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48124: BlockingRecv error: recv got EOF from 127.0.0.1:48124 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 01:17:30.295615 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.293414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.20.151.61:33289)
0812 01:17:30.293673 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.293680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.294208 (+ 528us) server_negotiation.cc:407] Connection header received
0812 01:17:30.294426 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.294431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.294503 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.294589 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.295411 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48132: BlockingRecv error: recv got EOF from 127.0.0.1:48132 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260812 01:17:30.311434 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.309233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.20.151.61:33289)
0812 01:17:30.309453 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.309460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.309992 (+ 532us) server_negotiation.cc:407] Connection header received
0812 01:17:30.310199 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.310204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.310279 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.310365 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.311149 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48134: BlockingRecv error: recv got EOF from 127.0.0.1:48134 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260812 01:17:30.333129 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.330924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.20.151.61:33289)
0812 01:17:30.331140 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.331148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.331678 (+ 530us) server_negotiation.cc:407] Connection header received
0812 01:17:30.331885 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.331889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.331985 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.332073 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.332861 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48150: BlockingRecv error: recv got EOF from 127.0.0.1:48150 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 01:17:30.370802 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.368691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.20.151.61:33289)
0812 01:17:30.368910 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.368917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.369260 (+ 343us) server_negotiation.cc:407] Connection header received
0812 01:17:30.369422 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.369428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.369545 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.369681 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.370517 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48152: BlockingRecv error: recv got EOF from 127.0.0.1:48152 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260812 01:17:30.440193 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.438216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.20.151.61:33289)
0812 01:17:30.438435 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.438442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.438694 (+ 252us) server_negotiation.cc:407] Connection header received
0812 01:17:30.438897 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.438904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.439023 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.439152 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.439956 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48156: BlockingRecv error: recv got EOF from 127.0.0.1:48156 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 01:17:30.573601 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:30.571686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.20.151.61:33289)
0812 01:17:30.571860 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.571865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.572327 (+ 462us) server_negotiation.cc:407] Connection header received
0812 01:17:30.572457 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.572463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.572549 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.572651 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:30.573349 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48168: BlockingRecv error: recv got EOF from 127.0.0.1:48168 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 01:17:30.650769 28074 catalog_manager.cc:1259] Loaded cluster ID: c3090ad5e8a24ab7a6890bf1c14a00e0
I20260812 01:17:30.650884 28074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7: loading cluster ID for follower catalog manager: success
I20260812 01:17:30.653398 28074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7: acquiring CA information for follower catalog manager: success
I20260812 01:17:30.655223 28074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bfa288741376486489f54a2a0d2ef2b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:17:30.737982 28089 catalog_manager.cc:1259] Loaded cluster ID: c3090ad5e8a24ab7a6890bf1c14a00e0
I20260812 01:17:30.738088 28089 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d: loading cluster ID for follower catalog manager: success
I20260812 01:17:30.740694 28089 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d: acquiring CA information for follower catalog manager: success
I20260812 01:17:30.742625 28089 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 70d7787b689c4487a442a03c3384778d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:17:30.839381 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:30.833655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.20.151.61:33289)
0812 01:17:30.833896 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.833904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.834077 (+ 173us) server_negotiation.cc:407] Connection header received
0812 01:17:30.834303 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.834308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.834472 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.834591 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:17:30.835028 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:30.835758 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:30.837099 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:30.837505 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:30.837628 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:30.838134 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:17:30.838718 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:17:30.838872 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:30.838973 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 01:17:30.839059 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 01:17:30.851646 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:30.845793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.20.151.60:35047)
0812 01:17:30.846318 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.846324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.846343 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:17:30.846462 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.846466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.846622 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.846786 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:17:30.847208 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:30.847983 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:30.849412 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:30.849840 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:30.849952 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:30.850440 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:17:30.850979 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:17:30.851132 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:30.851209 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 01:17:30.851322 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":261,"threads_started":1}
I20260812 01:17:30.865407 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:30.858417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.20.151.61:33289)
0812 01:17:30.858604 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.858611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.859402 (+ 791us) server_negotiation.cc:407] Connection header received
0812 01:17:30.859494 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.859500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.859587 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.859687 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:17:30.860279 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:30.861334 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:30.863330 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:30.863977 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:30.864118 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:30.864646 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:17:30.864788 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:17:30.864863 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:30.865040 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 01:17:30.865124 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 01:17:30.866473 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:30.859304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.20.151.60:35047)
0812 01:17:30.859923 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.859928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.860503 (+ 575us) server_negotiation.cc:407] Connection header received
0812 01:17:30.860658 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.860664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.860757 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.860853 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:17:30.861392 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:30.862435 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:30.864506 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:30.865027 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:30.865160 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:30.865732 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:17:30.865855 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:17:30.865914 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:30.866138 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 01:17:30.866221 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497}
I20260812 01:17:30.868593 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:30.858434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38598 (local address 127.20.151.62:44155)
0812 01:17:30.859504 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:30.859510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:30.862041 (+ 2531us) server_negotiation.cc:407] Connection header received
0812 01:17:30.862257 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:30.862263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:30.862478 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:30.862655 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:17:30.863732 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:30.864626 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:30.866488 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:30.867043 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:30.867165 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:30.867669 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:17:30.868012 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:17:30.868096 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:30.868219 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 01:17:30.868289 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":831,"threads_started":1}
I20260812 01:17:30.878796 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27682
I20260812 01:17:30.879895 27779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:17:30.997880 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 27682
I20260812 01:17:31.023947 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27796
I20260812 01:17:31.025154 27901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:17:31.139828 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 27796
I20260812 01:17:31.168071 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 27923
I20260812 01:17:31.169265 28037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:17:31.278545 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 27923
2026-08-12T01:17:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5040 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:17:31 dist-test-slave-21b9 krb5kdc[28278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:17:31 dist-test-slave-21b9 krb5kdc[28278](info): set up 2 sockets
Aug 12 01:17:31 dist-test-slave-21b9 krb5kdc[28278](info): commencing operation
krb5kdc: starting...
W20260812 01:17:32.593196 21084 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.004s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/test-user.keytab.
Aug 12 01:17:32 dist-test-slave-21b9 krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:17:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:17:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
Entry for principal kudu/127.20.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab.
I20260812 01:17:32.975363 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:37303
--webserver_interface=127.20.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:37303,127.20.151.61:38153,127.20.151.60:41395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:17:33.329846 28309 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:17:33.330192 28309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:33.330267 28309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:33.330363 28309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:33.337740 28309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:33.337841 28309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:33.337896 28309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:33.337949 28309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:33.338011 28309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:33.348157 28309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:37303,127.20.151.61:38153,127.20.151.60:41395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.62.keytab
--principal=kudu/127.20.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.62:37303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:33.349460 28309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:33.350770 28309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:33.362717 28316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:33.362762 28322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:33.362783 28319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:33.364061 28309 server_base.cc:1034] running on GCE node
I20260812 01:17:33.364836 28309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:33.366200 28309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:33.367401 28309 hybrid_clock.cc:648] HybridClock initialized: now 1786497453367316 us; error 49 us; skew 500 ppm
I20260812 01:17:33.367764 28309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:33 dist-test-slave-21b9 krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497453, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:33.374342 28309 init.cc:377] Logged in from keytab as kudu/127.20.151.62@KRBTEST.COM (short username kudu)
I20260812 01:17:33.375844 28309 webserver.cc:460] Webserver started at http://127.20.151.62:34717/ using document root <none> and password file <none>
I20260812 01:17:33.376530 28309 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:33.376622 28309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:33.376919 28309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:33.379894 28309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/data/instance:
uuid: "98e77bc6ed4e453783f16868c93121c7"
format_stamp: "Formatted at 2026-08-12 01:17:33 on dist-test-slave-21b9"
server_key: "be16af17d549695e9366719261c95b85"
server_key_iv: "d8ecfd2611e48c8c1cb367bfc512d2a2"
server_key_version: "encryptionkey@0"
I20260812 01:17:33.380662 28309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/wal/instance:
uuid: "98e77bc6ed4e453783f16868c93121c7"
format_stamp: "Formatted at 2026-08-12 01:17:33 on dist-test-slave-21b9"
server_key: "be16af17d549695e9366719261c95b85"
server_key_iv: "d8ecfd2611e48c8c1cb367bfc512d2a2"
server_key_version: "encryptionkey@0"
I20260812 01:17:33.385877 28309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:17:33.389537 28332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:33.390854 28309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:17:33.391013 28309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/wal
uuid: "98e77bc6ed4e453783f16868c93121c7"
format_stamp: "Formatted at 2026-08-12 01:17:33 on dist-test-slave-21b9"
server_key: "be16af17d549695e9366719261c95b85"
server_key_iv: "d8ecfd2611e48c8c1cb367bfc512d2a2"
server_key_version: "encryptionkey@0"
I20260812 01:17:33.391207 28309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:33.416556 28309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:33.420868 28309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:33.421111 28309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:33.439951 28309 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.62:37303
I20260812 01:17:33.440006 28418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.62:37303 every 8 connection(s)
I20260812 01:17:33.441323 28309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/data/info.pb
I20260812 01:17:33.443542 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 28309
I20260812 01:17:33.443828 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-0/wal/instance
I20260812 01:17:33.444437 21084 external_mini_cluster.cc:1428] Setting key 943c853dff634374b94c5bb84be371af
I20260812 01:17:33.445745 28421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:33.454658 28421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:17:33 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497453, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:17:33.477921 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.459620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.20.151.62:37303)
0812 01:17:33.460280 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:33.460290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:33.460331 (+ 41us) server_negotiation.cc:407] Connection header received
0812 01:17:33.462120 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:33.462127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:33.462388 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:33.462591 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:33.463970 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:33.465187 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:33.466571 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:33.467116 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:33.467328 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:33.470997 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:33.471006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:33.471019 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:33.471053 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:33.473755 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:33.474605 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:33.474612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:33.474614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:33.474764 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:33.475166 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:33.475169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:33.475171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:33.475726 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:33.475868 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:33.476498 (+ 630us) server_negotiation.cc:299] Negotiation successful
0812 01:17:33.476784 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":490,"thread_start_us":338,"threads_started":1}
I20260812 01:17:33.477960 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.458425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:37303 (local address 127.0.0.1:54204)
0812 01:17:33.459315 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:33.459365 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:33.461725 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:33.462632 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:33.462639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:33.463413 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:33.463768 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:33.463786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:33.465337 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:33.465341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:33.466430 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:33.466443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:33.467372 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:33.467379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:33.467624 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:33.468295 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:33.468319 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:33.470757 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:33.473902 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:33.473918 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:33.473933 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:33.474473 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:33.474868 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:33.474872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:33.474875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:33.475079 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:33.475863 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:33.475871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:33.476117 (+ 246us) client_negotiation.cc:769] Sending connection context
0812 01:17:33.476382 (+ 265us) client_negotiation.cc:240] Negotiation successful
0812 01:17:33.476959 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":86,"thread_start_us":334,"threads_started":1}
I20260812 01:17:33.481210 28421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.151.61@KRBTEST.COM; defaulting to no policy
I20260812 01:17:33.482255 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.481875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35254)
0812 01:17:33.482040 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:33.482150 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 01:17:33.482613 28334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
I20260812 01:17:33.484292 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.483968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35266)
0812 01:17:33.484099 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:33.484170 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.61@KRBTEST.COM" created.
W20260812 01:17:33.484498 28421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:38153: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab.
Entry for principal kudu/127.20.151.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab.
I20260812 01:17:33.525136 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.61:38153
--webserver_interface=127.20.151.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab
--principal=kudu/127.20.151.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:37303,127.20.151.61:38153,127.20.151.60:41395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:17:33.533857 28421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } attempt: 1
I20260812 01:17:33.535053 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.534576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35270)
0812 01:17:33.534788 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:33.534906 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 01:17:33.536137 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.535791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35278)
0812 01:17:33.535931 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:33.535999 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 01:17:33.536384 28421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:38153: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
I20260812 01:17:33.604661 28421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } attempt: 2
I20260812 01:17:33.606060 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.605459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35282)
0812 01:17:33.605742 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:33.605864 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 01:17:33.607129 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.606718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35298)
0812 01:17:33.606860 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:33.606967 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 01:17:33.607364 28421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:38153: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
I20260812 01:17:33.698666 28421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } attempt: 3
I20260812 01:17:33.700011 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.699442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35304)
0812 01:17:33.699713 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:33.699833 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 01:17:33.701148 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.700730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35310)
0812 01:17:33.700886 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:33.700976 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 01:17:33.701359 28421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:38153: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
I20260812 01:17:33.844647 28421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } attempt: 4
I20260812 01:17:33.846061 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.845367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35316)
0812 01:17:33.845756 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:33.845878 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260812 01:17:33.847221 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:33.846797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35332)
0812 01:17:33.846949 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:33.847055 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 01:17:33.847435 28421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.61:38153: Network error: Client connection negotiation failed: client connection to 127.20.151.61:38153: connect: Connection refused (error 111)
W20260812 01:17:33.883399 28433 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:17:33.883752 28433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:33.883857 28433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:33.883968 28433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:33.891459 28433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:33.891553 28433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:33.891645 28433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:33.891709 28433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:33.891794 28433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:33.901947 28433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:37303,127.20.151.61:38153,127.20.151.60:41395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.61.keytab
--principal=kudu/127.20.151.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.61:38153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.151.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:33.903303 28433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:33.904691 28433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:33.916647 28438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:33.916723 28439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:33.916879 28441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:33.917727 28433 server_base.cc:1034] running on GCE node
I20260812 01:17:33.918514 28433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:33.919986 28433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:33.921200 28433 hybrid_clock.cc:648] HybridClock initialized: now 1786497453921120 us; error 50 us; skew 500 ppm
I20260812 01:17:33.921536 28433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:33 dist-test-slave-21b9 krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497453, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:33.924814 28433 init.cc:377] Logged in from keytab as kudu/127.20.151.61@KRBTEST.COM (short username kudu)
I20260812 01:17:33.926424 28433 webserver.cc:460] Webserver started at http://127.20.151.61:40589/ using document root <none> and password file <none>
I20260812 01:17:33.927165 28433 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:33.927261 28433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:33.927621 28433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:33.930703 28433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/data/instance:
uuid: "f6e80b3cac2d4d02b9891c67dd26de66"
format_stamp: "Formatted at 2026-08-12 01:17:33 on dist-test-slave-21b9"
server_key: "923335115188bdfa16915797bbddd07b"
server_key_iv: "5fa1dd408d8b0641006c14c94dd76ca6"
server_key_version: "encryptionkey@0"
I20260812 01:17:33.931536 28433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/wal/instance:
uuid: "f6e80b3cac2d4d02b9891c67dd26de66"
format_stamp: "Formatted at 2026-08-12 01:17:33 on dist-test-slave-21b9"
server_key: "923335115188bdfa16915797bbddd07b"
server_key_iv: "5fa1dd408d8b0641006c14c94dd76ca6"
server_key_version: "encryptionkey@0"
I20260812 01:17:33.936729 28433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260812 01:17:33.940237 28452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:33.941583 28433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:17:33.941743 28433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/wal
uuid: "f6e80b3cac2d4d02b9891c67dd26de66"
format_stamp: "Formatted at 2026-08-12 01:17:33 on dist-test-slave-21b9"
server_key: "923335115188bdfa16915797bbddd07b"
server_key_iv: "5fa1dd408d8b0641006c14c94dd76ca6"
server_key_version: "encryptionkey@0"
I20260812 01:17:33.941964 28433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:33.966321 28433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:33.970890 28433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:33.971174 28433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:33.990269 28433 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.61:38153
I20260812 01:17:33.990294 28539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.61:38153 every 8 connection(s)
I20260812 01:17:33.991714 28433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/data/info.pb
I20260812 01:17:33.992306 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 28433
I20260812 01:17:33.992496 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-1/wal/instance
I20260812 01:17:33.993100 21084 external_mini_cluster.cc:1428] Setting key b8191f3b7ba297d03cbb7dbd91f7fa51
I20260812 01:17:33.995880 28540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:34.002775 28540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497453, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:17:34.026819 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.005683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.20.151.62:37303)
0812 01:17:34.006212 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.006217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.007331 (+ 1114us) server_negotiation.cc:407] Connection header received
0812 01:17:34.009760 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.009766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.009858 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.009957 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.012097 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.013139 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.015482 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.015922 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.016026 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.020487 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.020493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.020496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.020547 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.023297 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.024350 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.024353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.024356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.024486 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.024940 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.024943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.024945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.025237 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.025371 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.026343 (+ 972us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.026531 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":266,"threads_started":1}
I20260812 01:17:34.028338 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.006334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:37303 (local address 127.0.0.1:54216)
0812 01:17:34.007137 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.007182 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:34.009413 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:34.010246 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:34.010259 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:34.011042 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:34.011858 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.011884 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.013387 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.013393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.015324 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.015342 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.016171 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.016178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.016538 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.017283 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:34.017311 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:34.020196 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:34.023494 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.023504 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.023523 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.024234 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.024589 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.024593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.024596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.024846 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.025379 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:34.025389 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:34.025963 (+ 574us) client_negotiation.cc:769] Sending connection context
0812 01:17:34.026264 (+ 301us) client_negotiation.cc:240] Negotiation successful
0812 01:17:34.026799 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":319,"threads_started":1}
I20260812 01:17:34.030318 28540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.151.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.151.60@KRBTEST.COM" created.
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497453, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
I20260812 01:17:34.045403 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.031131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35348)
0812 01:17:34.031383 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.031404 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:34.031519 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:34.032726 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:34.032729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:34.033114 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:34.033261 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.033269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.034822 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.034826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.035810 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.035823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.036777 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.036782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.036977 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.037558 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:34.037574 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:34.039801 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:34.042865 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.042870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.042875 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.043248 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.043653 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.043658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.043662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.043790 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.044407 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:34.044411 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:34.044552 (+ 141us) client_negotiation.cc:769] Sending connection context
0812 01:17:34.044666 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 01:17:34.044949 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 01:17:34.045467 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.031404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.20.151.61:38153)
0812 01:17:34.032176 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.032186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.032221 (+ 35us) server_negotiation.cc:407] Connection header received
0812 01:17:34.032280 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.032289 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.032560 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.032724 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.033436 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.034696 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.035950 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.036605 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.036714 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.040004 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.040017 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.040032 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.040067 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.042741 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.043393 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.043400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.043403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.043532 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.043895 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.043898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.043900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.044263 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.044475 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.044824 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.045058 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":576,"thread_start_us":403,"threads_started":1}
I20260812 01:17:34.048147 28540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:17:34.049317 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.048823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52284)
0812 01:17:34.048977 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.049135 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 01:17:34.049562 28458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
I20260812 01:17:34.051093 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.050765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52290)
0812 01:17:34.050908 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.050984 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 01:17:34.051331 28540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:41395: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.151.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab.
Entry for principal kudu/127.20.151.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab.
I20260812 01:17:34.083575 21084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWp05DU/build/asan/bin/kudu
/tmp/dist-test-taskWp05DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.60:41395
--webserver_interface=127.20.151.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab
--principal=kudu/127.20.151.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.151.20:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.151.62:37303,127.20.151.61:38153,127.20.151.60:41395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:17:34.100663 28540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } attempt: 1
I20260812 01:17:34.101962 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.101475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52294)
0812 01:17:34.101682 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.101798 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 01:17:34.103016 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.102656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52296)
0812 01:17:34.102808 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.102901 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 01:17:34.103230 28540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:41395: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
I20260812 01:17:34.146667 28421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } attempt: 5
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497453, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
I20260812 01:17:34.164559 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.147533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.20.151.61:38153)
0812 01:17:34.147738 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.147744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.147851 (+ 107us) server_negotiation.cc:407] Connection header received
0812 01:17:34.148039 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.148045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.148168 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.148298 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.149352 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.150439 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.152204 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.152920 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.153062 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.157544 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.157552 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.157557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.157604 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.161349 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.162276 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.162283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.162286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.162446 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.162889 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.162894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.162897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.163354 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.163516 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.163991 (+ 475us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.164245 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 01:17:34.164569 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.147451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35362)
0812 01:17:34.147743 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.147762 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:34.147903 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:34.148354 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:34.148359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:34.148904 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:34.149177 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.149192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.150590 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.150597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.152041 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.152058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.153098 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.153105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.153365 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.154458 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:34.154485 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:34.157307 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:34.161527 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.161534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.161538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.162128 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.162569 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.162574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.162578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.162764 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.163472 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:34.163477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:34.163678 (+ 201us) client_negotiation.cc:769] Sending connection context
0812 01:17:34.163839 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 01:17:34.164184 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260812 01:17:34.166129 28421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:17:34.167375 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.166844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52302)
0812 01:17:34.167058 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.167167 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 01:17:34.168458 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.168079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52312)
0812 01:17:34.168235 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.168337 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 01:17:34.168738 28421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:41395: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
I20260812 01:17:34.171460 28540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } attempt: 2
I20260812 01:17:34.172533 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.172046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52322)
0812 01:17:34.172239 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.172345 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 01:17:34.173537 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.173159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52334)
0812 01:17:34.173306 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.173394 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 01:17:34.173791 28540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:41395: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
I20260812 01:17:34.220007 28421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } attempt: 1
I20260812 01:17:34.221544 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.220957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52342)
0812 01:17:34.221203 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.221322 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 01:17:34.222872 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.222372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52358)
0812 01:17:34.222551 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.222702 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 01:17:34.223130 28421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:41395: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
I20260812 01:17:34.265025 28540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } attempt: 3
I20260812 01:17:34.266556 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.265898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52364)
0812 01:17:34.266203 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.266331 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 01:17:34.267952 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.267378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52372)
0812 01:17:34.267679 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.267780 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260812 01:17:34.268250 28540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:41395: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
I20260812 01:17:34.291380 28421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } attempt: 2
I20260812 01:17:34.292888 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.292240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52384)
0812 01:17:34.292516 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.292661 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 01:17:34.294237 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.293736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52398)
0812 01:17:34.293913 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.294061 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 01:17:34.294529 28421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:41395: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
I20260812 01:17:34.400787 28421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } attempt: 3
I20260812 01:17:34.402076 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.401543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52414)
0812 01:17:34.401780 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.401876 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 01:17:34.403245 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.402812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52420)
0812 01:17:34.403040 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.403128 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260812 01:17:34.403512 28421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:41395: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
I20260812 01:17:34.411531 28540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } attempt: 4
I20260812 01:17:34.412817 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.412291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52434)
0812 01:17:34.412541 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.412646 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 01:17:34.413879 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.413519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52438)
0812 01:17:34.413677 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.413754 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 01:17:34.414134 28540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.151.60:41395: Network error: Client connection negotiation failed: client connection to 127.20.151.60:41395: connect: Connection refused (error 111)
W20260812 01:17:34.468045 28553 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:17:34.468394 28553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:17:34.468473 28553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:17:34.468564 28553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:17:34.476042 28553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:17:34.476143 28553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:17:34.476209 28553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:17:34.476260 28553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:17:34.476321 28553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:17:34.486310 28553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.151.20:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.151.62:37303,127.20.151.61:38153,127.20.151.60:41395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/krb5kdc/kudu_127.20.151.60.keytab
--principal=kudu/127.20.151.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.151.60:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.151.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:17:34.487638 28553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:17:34.488991 28553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:17:34.501281 28564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:34.501252 28563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:17:34.501367 28566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:17:34.502310 28553 server_base.cc:1034] running on GCE node
I20260812 01:17:34.503160 28553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:17:34.504678 28553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:17:34.505892 28553 hybrid_clock.cc:648] HybridClock initialized: now 1786497454505820 us; error 46 us; skew 500 ppm
I20260812 01:17:34.506196 28553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497454, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:17:34.509394 28553 init.cc:377] Logged in from keytab as kudu/127.20.151.60@KRBTEST.COM (short username kudu)
I20260812 01:17:34.510952 28553 webserver.cc:460] Webserver started at http://127.20.151.60:36073/ using document root <none> and password file <none>
I20260812 01:17:34.511761 28553 fs_manager.cc:359] Metadata directory not provided
I20260812 01:17:34.511897 28553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:17:34.512230 28553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:17:34.515295 28553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/data/instance:
uuid: "a209122d5e68418e96cc58d19ddd4e38"
format_stamp: "Formatted at 2026-08-12 01:17:34 on dist-test-slave-21b9"
server_key: "3d45fc4a59a181c5efd844da7f43c3d3"
server_key_iv: "b3159914b619aadc425624718cfd9691"
server_key_version: "encryptionkey@0"
I20260812 01:17:34.516098 28553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/wal/instance:
uuid: "a209122d5e68418e96cc58d19ddd4e38"
format_stamp: "Formatted at 2026-08-12 01:17:34 on dist-test-slave-21b9"
server_key: "3d45fc4a59a181c5efd844da7f43c3d3"
server_key_iv: "b3159914b619aadc425624718cfd9691"
server_key_version: "encryptionkey@0"
I20260812 01:17:34.521266 28553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 01:17:34.525014 28580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:17:34.526571 28553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:17:34.526733 28553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/wal
uuid: "a209122d5e68418e96cc58d19ddd4e38"
format_stamp: "Formatted at 2026-08-12 01:17:34 on dist-test-slave-21b9"
server_key: "3d45fc4a59a181c5efd844da7f43c3d3"
server_key_iv: "b3159914b619aadc425624718cfd9691"
server_key_version: "encryptionkey@0"
I20260812 01:17:34.526937 28553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:17:34.544049 28553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:17:34.548378 28553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:17:34.548650 28553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:17:34.565872 28553 rpc_server.cc:307] RPC server started. Bound to: 127.20.151.60:41395
I20260812 01:17:34.565891 28670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.151.60:41395 every 8 connection(s)
I20260812 01:17:34.567217 28553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/data/info.pb
I20260812 01:17:34.571133 28671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:17:34.575704 21084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWp05DU/build/asan/bin/kudu as pid 28553
I20260812 01:17:34.575886 21084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWp05DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786497066589499-21084-0/minicluster-data/master-2/wal/instance
I20260812 01:17:34.576323 21084 external_mini_cluster.cc:1428] Setting key 176fd660738babefc5f26ef05569e9f9
I20260812 01:17:34.578205 28671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:17:34.580744 28421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } attempt: 4
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497454, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497453, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.62@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
I20260812 01:17:34.599121 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.581371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.20.151.62:37303)
0812 01:17:34.581966 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.581972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.582693 (+ 721us) server_negotiation.cc:407] Connection header received
0812 01:17:34.582754 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.582759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.582829 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.582931 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.584753 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.585514 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.587438 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.587854 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.587964 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.592467 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.592474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.592479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.592533 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.595695 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.596651 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.596655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.596658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.596778 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.597227 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.597230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.597232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.597577 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.597709 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.598451 (+ 742us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.598702 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":332,"threads_started":1}
I20260812 01:17:34.599844 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.581369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52458)
0812 01:17:34.581527 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.581546 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:34.581661 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:34.583920 (+ 2259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:34.583925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:34.584404 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:34.584566 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.584579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.586396 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.586401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.587314 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.587329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.588735 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.588741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.588922 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.589457 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:34.589479 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:34.592521 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:34.596956 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.596962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.596965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.597490 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.598006 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.598011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.598014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.598205 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.598949 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:34.598953 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:34.599129 (+ 176us) client_negotiation.cc:769] Sending connection context
0812 01:17:34.599260 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 01:17:34.599555 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 01:17:34.600471 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.581469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.62:37303 (local address 127.0.0.1:54228)
0812 01:17:34.582352 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.582392 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:34.582568 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:34.583218 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:34.583225 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:34.583998 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:34.584575 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.584599 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.585671 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.585674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.587264 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.587307 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.588136 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.588144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.588543 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.589395 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:34.589427 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:34.592247 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:34.595816 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.595829 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.595843 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.596476 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.596887 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.596892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.596896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.597109 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.597706 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:34.597714 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:34.598091 (+ 377us) client_negotiation.cc:769] Sending connection context
0812 01:17:34.598355 (+ 264us) client_negotiation.cc:240] Negotiation successful
0812 01:17:34.598927 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":114,"thread_start_us":509,"threads_started":1}
I20260812 01:17:34.601183 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.582757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.20.151.60:41395)
0812 01:17:34.583596 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.583603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.583635 (+ 32us) server_negotiation.cc:407] Connection header received
0812 01:17:34.583685 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.583690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.583753 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.583859 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.584743 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.586235 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.587455 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.588553 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.589163 (+ 610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.592713 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.592721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.592735 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.592763 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.596793 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.597678 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.597687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.597690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.597866 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.598337 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.598342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.598345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.598794 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.600274 (+ 1480us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.600437 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.600731 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":335,"server-negotiator.queue_time_us":695,"thread_start_us":493,"threads_started":1}
I20260812 01:17:34.601532 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.578122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.20.151.60:41395)
0812 01:17:34.578970 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.578986 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.579026 (+ 40us) server_negotiation.cc:407] Connection header received
0812 01:17:34.581400 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.581804 (+ 404us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.582289 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.582590 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.584403 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.586148 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.587954 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.588546 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.588657 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.593412 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.593416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.593420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.593448 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.597166 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.598069 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.598074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.598077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.598193 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.598637 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.598641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.598644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.599025 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.600654 (+ 1629us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.600817 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.601080 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":626,"thread_start_us":382,"threads_started":1}
I20260812 01:17:34.603152 28671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:17:34.607903 28421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7: Bootstrap starting.
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497454, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
I20260812 01:17:34.611802 28421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:34.613103 28421 log.cc:824] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:34.616519 28421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7: No bootstrap required, opened a new log
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.62@KRBTEST.COM
I20260812 01:17:34.617938 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.604643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.20.151.61:38153)
0812 01:17:34.604819 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.604824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.604840 (+ 16us) server_negotiation.cc:407] Connection header received
0812 01:17:34.604890 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.604894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.604949 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.605033 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.605867 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.606544 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.607980 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.608615 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.608747 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.612248 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.612256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.612260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.612305 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.615201 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.615897 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.615901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.615904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.616057 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.616415 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.616419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.616421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.616750 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.616887 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.617318 (+ 431us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.617548 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:17:34.617956 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.603986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.61:38153 (local address 127.0.0.1:35364)
0812 01:17:34.604159 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.604180 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:34.604279 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:34.605092 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:34.605097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:34.605570 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:34.605720 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.605729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.606712 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.606718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.607817 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.607832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.608787 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.608793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.608992 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.609670 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:34.609694 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:34.612046 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:34.615367 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.615372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.615376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.615776 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.616168 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.616173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.616175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.616312 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.616865 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:34.616870 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:34.617034 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 01:17:34.617188 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 01:17:34.617498 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 01:17:34.619481 28671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:17:34.625422 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.605316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.20.151.62:37303)
0812 01:17:34.605481 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.605487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.607717 (+ 2230us) server_negotiation.cc:407] Connection header received
0812 01:17:34.607778 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.607782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.607882 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.608020 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.609453 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.610519 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.612241 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.612896 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.613013 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.618818 (+ 5805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.618824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.618832 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.618878 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.622242 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.622915 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.622920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.622924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.623091 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.623588 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.623594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.623597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.624009 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.624281 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.624853 (+ 572us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.625090 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:17:34.626964 28421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } }
I20260812 01:17:34.627409 28421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98e77bc6ed4e453783f16868c93121c7, State: Initialized, Role: FOLLOWER
I20260812 01:17:34.627967 28421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } }
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497454, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.60@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
I20260812 01:17:34.628971 28683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } } }
I20260812 01:17:34.629578 28683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:34.630456 28421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.151.61@KRBTEST.COM
I20260812 01:17:34.636003 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.620185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52460)
0812 01:17:34.620382 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.620401 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:34.620521 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:34.620977 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:34.620982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:34.621428 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:34.621641 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.621654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.622961 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.622966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.624283 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.624299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.625341 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.625347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.625534 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.626484 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:34.626506 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:34.629110 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:34.632982 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.632987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.632992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.633494 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.633938 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.633943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.633947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.634117 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.634806 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:34.634811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:34.634979 (+ 168us) client_negotiation.cc:769] Sending connection context
0812 01:17:34.635148 (+ 169us) client_negotiation.cc:240] Negotiation successful
0812 01:17:34.635538 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 01:17:34.636795 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.620529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.20.151.60:41395)
0812 01:17:34.620687 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.620693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.620710 (+ 17us) server_negotiation.cc:407] Connection header received
0812 01:17:34.620755 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.620759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.620825 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.620923 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.621795 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.622844 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.624427 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.625045 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.625172 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.629315 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.629321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.629325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.629361 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.632811 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.633629 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.633635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.633638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.633803 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.634235 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.634241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.634244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.634692 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.634850 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.635291 (+ 441us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.636435 (+ 1144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":880,"server-negotiator.queue_time_us":61}
I20260812 01:17:34.642904 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.627138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.20.151.61:38153)
0812 01:17:34.627327 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.627334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.627620 (+ 286us) server_negotiation.cc:407] Connection header received
0812 01:17:34.627676 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.627679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.627739 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.627816 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.629336 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.630306 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.631865 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.632312 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.632425 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.636888 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.636897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.636903 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.636950 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.639767 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.640609 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.640613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.640615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.640760 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.641156 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.641162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.641165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.641470 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.641829 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.642333 (+ 504us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.642570 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:17:34.643050 28671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38: Bootstrap starting.
I20260812 01:17:34.649060 28671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:34.650517 28671 log.cc:824] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:34.651393 21084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:17:34.654174 28671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38: No bootstrap required, opened a new log
I20260812 01:17:34.654428 28380 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "98e77bc6ed4e453783f16868c93121c7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54230
I20260812 01:17:34.654929 28380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 01:17:34.655153 28380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:34.659296 28380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } }
I20260812 01:17:34.662920 28380 leader_election.cc:288] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [CANDIDATE]: Term 1 election: Requested vote from peers f6e80b3cac2d4d02b9891c67dd26de66 (127.20.151.61:38153), a209122d5e68418e96cc58d19ddd4e38 (127.20.151.60:41395)
I20260812 01:17:34.650015 28309 master_runner.cc:186] Error getting master registration for 127.20.151.61:38153: OK, instance_id { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" instance_seqno: 1786497453984673 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:17:34.663638 28309 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 01:17:34.666009 28671 raft_consensus.cc:348] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } }
I20260812 01:17:34.666676 28671 raft_consensus.cc:729] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a209122d5e68418e96cc58d19ddd4e38, State: Initialized, Role: FOLLOWER
I20260812 01:17:34.667402 28671 consensus_queue.cc:260] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } }
I20260812 01:17:34.669071 28717 sys_catalog.cc:455] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } } }
I20260812 01:17:34.670221 28671 sys_catalog.cc:564] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:34.670327 28625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98e77bc6ed4e453783f16868c93121c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a209122d5e68418e96cc58d19ddd4e38"
I20260812 01:17:34.670809 28625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [term 0 FOLLOWER]: Advancing to term 1
W20260812 01:17:34.672601 28705 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:17:34.672739 28705 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:17:34.673895 28625 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98e77bc6ed4e453783f16868c93121c7 in term 1.
W20260812 01:17:34.674116 28340 tablet.cc:2369] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:17:34.674674 28337 leader_election.cc:302] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98e77bc6ed4e453783f16868c93121c7, a209122d5e68418e96cc58d19ddd4e38; no voters:
I20260812 01:17:34.675089 28683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:17:34.670122 28717 sys_catalog.cc:458] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:34.675776 28683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [term 1 LEADER]: Becoming Leader. State: Replica: 98e77bc6ed4e453783f16868c93121c7, State: Running, Role: LEADER
I20260812 01:17:34.676326 28683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } }
I20260812 01:17:34.684505 28719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98e77bc6ed4e453783f16868c93121c7. Latest consensus state: current_term: 1 leader_uuid: "98e77bc6ed4e453783f16868c93121c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } } }
I20260812 01:17:34.684868 28719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:34.685737 28730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:17:34.688175 28730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:17:34.700294 28625 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [term 1 FOLLOWER]: Refusing update from remote peer 98e77bc6ed4e453783f16868c93121c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:17:34.701583 28719 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:17:34.713353 28540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } attempt: 5
Aug 12 01:17:34 dist-test-slave-21b9 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786497453, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.151.61@KRBTEST.COM for kudu/127.20.151.60@KRBTEST.COM
I20260812 01:17:34.735173 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.714031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.151.60:41395 (local address 127.0.0.1:52472)
0812 01:17:34.714290 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0812 01:17:34.714309 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:17:34.714454 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:17:34.714974 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:17:34.714980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:17:34.715591 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:17:34.715831 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.715845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.717440 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.717445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.718916 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:17:34.718936 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.720345 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.720353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:17:34.720655 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.721388 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:17:34.721413 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:17:34.724439 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:17:34.731981 (+ 7542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.731986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.731990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.732607 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.733069 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:17:34.733074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:17:34.733077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:17:34.733255 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:17:34.734032 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:17:34.734038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:17:34.734212 (+ 174us) client_negotiation.cc:769] Sending connection context
0812 01:17:34.734348 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 01:17:34.734707 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260812 01:17:34.736389 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.714092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.20.151.60:41395)
0812 01:17:34.714280 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.714287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.714369 (+ 82us) server_negotiation.cc:407] Connection header received
0812 01:17:34.714581 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.714588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.714680 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.714774 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.716028 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.717242 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.719120 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.719927 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.720048 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.724670 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.724676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.724681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.724722 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.731706 (+ 6984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.732773 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.732779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.732783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.732939 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.733417 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.733421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.733424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.733890 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.735671 (+ 1781us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.735817 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.736051 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 01:17:34.743135 28540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66: Bootstrap starting.
I20260812 01:17:34.747884 28540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66: Neither blocks nor log segments found. Creating new log.
I20260812 01:17:34.749369 28540 log.cc:824] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66: Log is configured to *not* fsync() on all Append() calls
I20260812 01:17:34.753611 28717 sys_catalog.cc:455] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98e77bc6ed4e453783f16868c93121c7. Latest consensus state: current_term: 1 leader_uuid: "98e77bc6ed4e453783f16868c93121c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } } }
I20260812 01:17:34.753940 28717 sys_catalog.cc:458] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:34.756421 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.702881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.20.151.62:37303)
0812 01:17:34.703077 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.703084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.723904 (+ 20820us) server_negotiation.cc:407] Connection header received
0812 01:17:34.724464 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.724470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.724563 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.724641 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.730216 (+ 5575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.731536 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.738590 (+ 7054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.739436 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.739554 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.744776 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.744783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.744788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.744850 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.748994 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.751192 (+ 2198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.751197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.751201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.751391 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.753451 (+ 2060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.753456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.753460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.753892 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.754161 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.755738 (+ 1577us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.756035 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:17:34.757390 28740 mvcc.cc:205] Tried to move back new op lower bound from 7317493574436327424 to 7317493574359097344. Current Snapshot: MvccSnapshot[applied={T|T < 7317493574436327424}]
I20260812 01:17:34.766649 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.707110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.20.151.61:38153)
0812 01:17:34.707324 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.707331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.727459 (+ 20128us) server_negotiation.cc:407] Connection header received
0812 01:17:34.727707 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.727713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.727802 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.727872 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.735168 (+ 7296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.736238 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.745376 (+ 9138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.746062 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.746178 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.754753 (+ 8575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.754761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.754766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.754806 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.758690 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.761632 (+ 2942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.761638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.761641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.761816 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.763444 (+ 1628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.763450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.763454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.763899 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.764150 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.766003 (+ 1853us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.766263 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 01:17:34.767642 28540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66: No bootstrap required, opened a new log
I20260812 01:17:34.769014 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:34.734003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.20.151.60:41395)
0812 01:17:34.734211 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.734216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:34.741938 (+ 7722us) server_negotiation.cc:407] Connection header received
0812 01:17:34.744739 (+ 2801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:34.744745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:34.744833 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:34.744901 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:34.750536 (+ 5635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.751678 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.757652 (+ 5974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:34.758362 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:34.758462 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:34.761830 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:17:34.761836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:17:34.761840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:17:34.761882 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:17:34.765710 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.766545 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.766550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.766554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.766707 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:17:34.767190 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:17:34.767196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:17:34.767199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:17:34.767610 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:17:34.767874 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:34.768457 (+ 583us) server_negotiation.cc:299] Negotiation successful
0812 01:17:34.768695 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 01:17:34.771099 28717 sys_catalog.cc:455] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98e77bc6ed4e453783f16868c93121c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } } }
I20260812 01:17:34.772279 28717 sys_catalog.cc:458] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:34.773152 28753 mvcc.cc:205] Tried to move back new op lower bound from 7317493574436327424 to 7317493574359097344. Current Snapshot: MvccSnapshot[applied={T|T < 7317493574436327424}]
I20260812 01:17:34.778467 28730 catalog_manager.cc:1347] Generated new cluster ID: 901592be56624136a724995b77f168d3
I20260812 01:17:34.778584 28730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:17:34.780756 28719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98e77bc6ed4e453783f16868c93121c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } } }
I20260812 01:17:34.781044 28719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:34.786556 28540 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } }
I20260812 01:17:34.786978 28540 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e80b3cac2d4d02b9891c67dd26de66, State: Initialized, Role: FOLLOWER
I20260812 01:17:34.787508 28540 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } }
I20260812 01:17:34.836583 28771 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } } }
I20260812 01:17:34.790145 28553 master_runner.cc:186] Error getting master registration for 127.20.151.61:38153: OK, instance_id { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" instance_seqno: 1786497453984673 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:17:34.837977 28771 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:34.838107 28540 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:17:34.839954 28498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:17:34.841519 28499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98e77bc6ed4e453783f16868c93121c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f6e80b3cac2d4d02b9891c67dd26de66"
I20260812 01:17:34.844729 28498 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [term 1 FOLLOWER]: Refusing update from remote peer 98e77bc6ed4e453783f16868c93121c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:17:34.845723 28683 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:17:34.848949 28499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98e77bc6ed4e453783f16868c93121c7 in term 1.
I20260812 01:17:34.852414 28553 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 01:17:34.856041 28762 catalog_manager.cc:1259] Loaded cluster ID: 901592be56624136a724995b77f168d3
I20260812 01:17:34.856184 28762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38: loading cluster ID for follower catalog manager: success
I20260812 01:17:34.856685 28730 catalog_manager.cc:1370] Generated new certificate authority record
W20260812 01:17:34.858331 28762 catalog_manager.cc:1576] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260812 01:17:34.858477 28762 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:17:34.858595 28730 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:17:34.872087 28730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7: Generated new TSK 0
I20260812 01:17:34.873378 28730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:17:34.886453 28797 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98e77bc6ed4e453783f16868c93121c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } } }
I20260812 01:17:34.886978 28797 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:34.887058 28771 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98e77bc6ed4e453783f16868c93121c7. Latest consensus state: current_term: 1 leader_uuid: "98e77bc6ed4e453783f16868c93121c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } } }
I20260812 01:17:34.887373 28771 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:17:34.890710 28719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98e77bc6ed4e453783f16868c93121c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98e77bc6ed4e453783f16868c93121c7" member_type: VOTER last_known_addr { host: "127.20.151.62" port: 37303 } } peers { permanent_uuid: "f6e80b3cac2d4d02b9891c67dd26de66" member_type: VOTER last_known_addr { host: "127.20.151.61" port: 38153 } } peers { permanent_uuid: "a209122d5e68418e96cc58d19ddd4e38" member_type: VOTER last_known_addr { host: "127.20.151.60" port: 41395 } } }
I20260812 01:17:34.891017 28719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e77bc6ed4e453783f16868c93121c7 [sys.catalog]: This master's current role is: LEADER
I20260812 01:17:34.894155 28803 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260812 01:17:35.004074 28676 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:34.988070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.20.151.62:37303)
0812 01:17:34.988472 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.988479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.000790 (+ 12311us) server_negotiation.cc:407] Connection header received
0812 01:17:35.002390 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.002397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.002485 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.002565 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.003834 (+ 1269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54238: BlockingRecv error: recv got EOF from 127.0.0.1:54238 (error 108)
Metrics: {"server-negotiator.queue_time_us":256}
W20260812 01:17:35.008061 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:34.988259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.20.151.61:38153)
0812 01:17:34.988625 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:34.988631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.005720 (+ 17089us) server_negotiation.cc:407] Connection header received
0812 01:17:35.006538 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.006544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.006649 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.006728 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.007826 (+ 1098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35400: BlockingRecv error: recv got EOF from 127.0.0.1:35400 (error 108)
Metrics: {"server-negotiator.queue_time_us":252}
W20260812 01:17:35.011592 28681 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:35.001407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.20.151.60:41395)
0812 01:17:35.001596 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:35.001602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.009420 (+ 7818us) server_negotiation.cc:407] Connection header received
0812 01:17:35.010095 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.010100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.010189 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.010254 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.011322 (+ 1068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52490: BlockingRecv error: recv got EOF from 127.0.0.1:52490 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260812 01:17:35.035192 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:35.032538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.20.151.61:38153)
0812 01:17:35.032738 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:35.032744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.033290 (+ 546us) server_negotiation.cc:407] Connection header received
0812 01:17:35.033542 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.033548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.033813 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.033940 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.034897 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35412: BlockingRecv error: recv got EOF from 127.0.0.1:35412 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 01:17:35.043628 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:35.040044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.20.151.61:38153)
0812 01:17:35.040287 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:35.040294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.041990 (+ 1696us) server_negotiation.cc:407] Connection header received
0812 01:17:35.042195 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.042201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.042305 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.042428 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.043394 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35414: BlockingRecv error: recv got EOF from 127.0.0.1:35414 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260812 01:17:35.059746 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:35.057257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.20.151.61:38153)
0812 01:17:35.057455 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:35.057461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.058038 (+ 577us) server_negotiation.cc:407] Connection header received
0812 01:17:35.058273 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.058280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.058402 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.058510 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.059479 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35420: BlockingRecv error: recv got EOF from 127.0.0.1:35420 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 01:17:35.069947 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:35.067528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.20.151.61:38153)
0812 01:17:35.067789 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:35.067795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.068326 (+ 531us) server_negotiation.cc:407] Connection header received
0812 01:17:35.068540 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.068546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.068661 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.068775 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.069662 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35426: BlockingRecv error: recv got EOF from 127.0.0.1:35426 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
W20260812 01:17:35.084105 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:35.081683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.20.151.61:38153)
0812 01:17:35.081886 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:35.081893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.082481 (+ 588us) server_negotiation.cc:407] Connection header received
0812 01:17:35.082710 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.082716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.082814 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.082925 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.083861 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35442: BlockingRecv error: recv got EOF from 127.0.0.1:35442 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 01:17:35.112414 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:35.109935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.20.151.61:38153)
0812 01:17:35.110144 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:35.110151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.110736 (+ 585us) server_negotiation.cc:407] Connection header received
0812 01:17:35.110947 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.110953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.111061 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.111172 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.112171 (+ 999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35452: BlockingRecv error: recv got EOF from 127.0.0.1:35452 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 01:17:35.156742 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:35.154110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.20.151.61:38153)
0812 01:17:35.154333 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:35.154341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.154948 (+ 607us) server_negotiation.cc:407] Connection header received
0812 01:17:35.155200 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.155206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.155346 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.155452 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.156456 (+ 1004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35458: BlockingRecv error: recv got EOF from 127.0.0.1:35458 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260812 01:17:35.226363 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:35.224396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.20.151.61:38153)
0812 01:17:35.224611 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:35.224617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.224877 (+ 260us) server_negotiation.cc:407] Connection header received
0812 01:17:35.225049 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.225055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.225181 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.225334 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.226118 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35464: BlockingRecv error: recv got EOF from 127.0.0.1:35464 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260812 01:17:35.360185 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:35.358245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.20.151.61:38153)
0812 01:17:35.358505 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:35.358513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.358708 (+ 195us) server_negotiation.cc:407] Connection header received
0812 01:17:35.358925 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.358930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.359004 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.359095 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.359914 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35466: BlockingRecv error: recv got EOF from 127.0.0.1:35466 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260812 01:17:35.621266 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:17:35.619665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.20.151.61:38153)
0812 01:17:35.619832 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:35.619837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:35.620072 (+ 235us) server_negotiation.cc:407] Connection header received
0812 01:17:35.620228 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:35.620232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:35.620305 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:35.620400 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:17:35.621055 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35470: BlockingRecv error: recv got EOF from 127.0.0.1:35470 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 01:17:35.862111 28762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38: acquiring CA information for follower catalog manager: success
I20260812 01:17:35.864069 28762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a209122d5e68418e96cc58d19ddd4e38: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:17:35.898723 28803 catalog_manager.cc:1259] Loaded cluster ID: 901592be56624136a724995b77f168d3
I20260812 01:17:35.898839 28803 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66: loading cluster ID for follower catalog manager: success
I20260812 01:17:35.901568 28803 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66: acquiring CA information for follower catalog manager: success
I20260812 01:17:35.903498 28803 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6e80b3cac2d4d02b9891c67dd26de66: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:17:36.148444 28916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:36.140166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.20.151.61:38153)
0812 01:17:36.140872 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:36.140880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:36.141237 (+ 357us) server_negotiation.cc:407] Connection header received
0812 01:17:36.141433 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:36.141439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:36.141673 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:36.141835 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:17:36.142233 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:36.143596 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:36.145253 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:36.146013 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:36.146172 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:36.146785 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:17:36.147707 (+ 922us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:17:36.147889 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:36.147950 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 01:17:36.148044 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":388,"threads_started":1}
I20260812 01:17:36.160988 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:36.155111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.20.151.60:41395)
0812 01:17:36.155799 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:36.155805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:36.155825 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:17:36.155934 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:36.155938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:36.156096 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:36.156255 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:17:36.156704 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:36.157442 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:36.158776 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:36.159188 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:36.159322 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:36.159867 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:17:36.160392 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:17:36.160552 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:36.160610 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 01:17:36.160699 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":432,"threads_started":1}
I20260812 01:17:36.187944 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:36.166695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.20.151.62:37303)
0812 01:17:36.178949 (+ 12254us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:36.178956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:36.179727 (+ 771us) server_negotiation.cc:407] Connection header received
0812 01:17:36.180130 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:36.180137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:36.180372 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:36.180539 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:17:36.181523 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:36.182761 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:36.184952 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:36.185572 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:36.185665 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:36.186450 (+ 785us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:17:36.186948 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:17:36.187025 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:36.187542 (+ 517us) server_negotiation.cc:299] Negotiation successful
0812 01:17:36.187636 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12102,"thread_start_us":12265,"threads_started":1}
I20260812 01:17:36.188272 28916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:36.167186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.20.151.61:38153)
0812 01:17:36.172981 (+ 5795us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:36.172988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:36.180002 (+ 7014us) server_negotiation.cc:407] Connection header received
0812 01:17:36.180506 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:36.180512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:36.180608 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:36.180688 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:17:36.181661 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:36.182750 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:36.185870 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:36.186566 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:36.186669 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:36.187384 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:17:36.187521 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:17:36.187587 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:36.187838 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 01:17:36.187930 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5628}
I20260812 01:17:36.190040 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:17:36.180084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.20.151.60:41395)
0812 01:17:36.180259 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 01:17:36.180265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:17:36.183698 (+ 3433us) server_negotiation.cc:407] Connection header received
0812 01:17:36.183864 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:17:36.183869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:17:36.183975 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:17:36.184068 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:17:36.184907 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:36.185919 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:36.187959 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:17:36.188529 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:17:36.188640 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:17:36.189274 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:17:36.189397 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:17:36.189461 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 01:17:36.189709 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 01:17:36.189793 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:17:36.194463 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 28309
I20260812 01:17:36.195520 28411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:17:36.309190 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 28309
W20260812 01:17:36.341148 28586 connection.cc:535] server connection from 127.0.0.1:52458 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 01:17:36.341527 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 28433
I20260812 01:17:36.342697 28534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:17:36.452766 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 28433
I20260812 01:17:36.482509 21084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWp05DU/build/asan/bin/kudu pid 28553
I20260812 01:17:36.483770 28661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:17:36.603006 21084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWp05DU/build/asan/bin/kudu with pid 28553
2026-08-12T01:17:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5328 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10369 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (389977 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST